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PREFACE 



At the 2000 Virtual Reference Desk 
Conference held in Seattle, Washington, a 
number of the speakers commented on the 
lack of assessment techniques and the limited 
number of statistics and measures that had 
been developed for digital reference services, 
and questioned the degree to which traditional 
assessment techniques could be used for 
digital reference services. While there was 
considerable progress with development of 
digital reference service, there was very 
limited work being done to assess digital 
reference services. 

Authors of this manual asserted to 
conference attendees that development of 
such methods, measures, statistics, and 
standards was essential. In addition, they 
challenged the audience to work together to 
support a research project that could assist in 
the ongoing assessment of digital reference - 
and ultimately result in some practical 
guidelines and procedures to conduct digital 
reference services assessment. 

Shortly after the conference, McClure and 
Lankes proposed a project that would result in 
a manual to describe specific techniques, 
statistics, and measures that could be used to 
assess digital reference services. The project, 
Assessing Quality in Digital Reference 
eventually was supported by a number of 
public, academic, and state libraries in both 
the United States and in the United Kingdom 
as well as a number of library consortia and 
other organizations (see acknowledgements). 
The original proposal and other project 
documents can be found at: 
http://quartz.syr.edu/quality/. The project 
began in March 2001 and was completed in 
May 2002. This manual is the primary 
product of the project. 

The process by which the study was 
conducted began with a review of the existing 
literature on evaluation of digital reference 
services. Next, the study team conducted site 



visits to various libraries participating in the 
study that were engaged in various aspects of 
digital reference services. These site visits 
identified issues and possible approaches that 
could be incorporated into the evaluation 
process. A summary of the findings, best 
practices, and key issues from the site visits 
can be found at: http://quartz.syr.edu/quality/ 
VRDSiteVisitsummaiy.pdf. 

Based on the literature review, the site 
visits, and the authors’ knowledge of the 
topic, a draft manual was developed that 
described a range of assessment techniques, 
statistics, and measures. The draft manual was 
reviewed by the project advisory committee 
and others, and then was revised by the study 
team. This version of the manual, then, was 
field tested by a number of libraries 
participating in the project. The results of the 
field test identified a number of areas where 
the manual could be improved. Based on 
these findings, the manual was re-drafted, 
reviewed, and appears here in its final 
version. 

Thus, the final version of the manual 
presented here represents the work of a 
number of individuals associated with the 
Assessing Quality in Digital Reference 
project (see acknowledgements). The authors 
see this manual as a first step to beginning an 
ongoing process to develop statistics, 
measures, and quality standards to assess and 
improve digital reference services. We hope 
that others will be able to build upon this 
manual and continue such assessment work in 
the future. 

Charles R. Mclure 
R. David Lankes 
Melissa Gross 
Beverly Choltco-Devlin 
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CHAPTER 1 1 



INTRODUCTION 



Background 

One of the fastest growing and most 
innovative services being developed by 
libraries today is digital reference. Such 
services refer to a network of expertise, 
intermediation and resources put at the 
disposal of a user seeking answers in an 
online/networked environment. A digital 
reference transaction occurs when a question 
is received electronically and responded to 
electronically (Bertot, McClure, & Ryan, 
2000, p. 12). While the development and 
implementation of digital reference services 
has received wide attention during the last 
several years, only limited attention has been 
given to the assessment of digital reference 
services. 

Many libraries are now providing digital 
reference service, either as an integrated 
component of their regular reference service, 
as a separate service, or as part of a 
collaborative consortium. Many other 
libraries are thinking about or are about to 
implement such services. Additionally, an 
increasing number of digital reference 
services (“AskA”) have been developed in the 
commercial, educational and non-profit 
sectors that are not directly affiliated with any 
specific library (e.g., AskJeeves, Internet 
Public Library, AskERIC). The focus of both 
library and non-library affiliated digital 
reference services has been to provide 
information to the user via electronic means. 
Many see the provision of digital reference 
service as a way of increasing access to 
library services in a manner that is not 
dependent on physical visits to the library. 



People have become increasingly 
comfortable utilizing and, indeed, relying on 
digital services as part of their way of life. For 
example, many people are now shopping, 
banking, and paying their bills online. They 
also communicate with others in their 
personal and business lives by using email or 
real time services such as online chat, instant 
message services, or video conferencing. 
People are also beginning to expect their 
libraries to provide some type of digital 
service. These services include access to the 
online catalog, the ability to place requests 
online, access to electronic resources and, of 
course, the provision of some type of digital 
reference service, which is the focus of this 
manual. 

The provision of digital reference service 
can take many forms and has developed in a 
variety of ways. In many cases, digital 
reference service, as provided by libraries of 
every type, has evolved naturally and does not 
exist separately from traditional reference 
service. Here, reference staff answer 
questions received both in traditional ways 
(face-to-face at the reference desk or via the 
phone) and digitally (most often via email or 
chat). They often also respond via a 
combination of traditional and electronic 
(digital) methods. In such situations, even a 
single reference transaction can be comprised 
of a combination of both traditional and 
digital methods of communication and 
response (including the provision of the 
answer). In these cases, the reference staff 
members are not specifically designated as 
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either traditional or digital reference 
librarians. 

In other library situations, a separate 
digital reference service has been developed 
and is offered to users. In such situations, the 
digital reference service has been designed to 
function as a separate service from the 
provision of traditional reference service. 
Personnel are often assigned to the digital 
reference service or a completely separate 
staff exists to provide strictly digital reference 
service. In many cases, the digital reference 
service is advertised and promoted as a 
separate library service, and the points of 
access to the service (and even in some cases, 
the provision of service) are not tied to the 
physical library facility. 

These points of access can include a 
separate digital reference web page within the 
library’s website, a separate email address, or 
the capacity for online chat, interactive video, 
or voice over IP sessions (which may also be 
accessed through the library’s digital 
reference website). Additionally, these digital 
library services are often available to the user 
during times when the physical library is not 
open. And of course these services can be 
provided regardless of the user’s physical 
location. Indeed, one of the main reasons that 
a library develops a separate digital reference 
service is to help meet a user’s needs, which 
may not match the library’s ability to provide 
reference service in a traditional way. 

Finally, there are a number of digital 
reference services that are not affiliated with 
any specific library. These are often called 
“AskA” services. AskERIC (http://www. 
askeric.org) and the Internet Public Library 
(http://www.ipl.org/ref/QUE/) are excellent 
examples of such a service. With these 
services, there is no physical place to which 
the user goes; all transactions with the service 
are conducted digitally. The service and 
staff’s sole purpose is to provide digital 
reference service. 



In each of these situations, digital 
reference service is being provided to some 
degree. In many cases reference transactions 
may be comprised of a combination of digital 
and more traditional methods of 
communication between the user and the 
reference service staff. For the purposes of 
this manual, however, a digital reference 
transaction is defined as one in which all 
communication between the user and staff is 
conducted electronically or digitally. Thus, a 
digital reference question is counted as such 
only if the entire transaction is conducted 
digitally; the question must be received 
digitally and all responses, including the 
answer, must be sent digitally. 

If a question is received digitally and 
answered traditionally or with a combination 
of traditional or digital means, then the 
question is counted under Measure 4 - 
Number of questions received digitally but not 
answered or responded to by completely 
digital means. These definitions of what 
constitutes a digital reference question and 
digital reference answer should not, in any 
way, be interpreted as a belief on the part of 
the authors that hybrid transactions are not on 
some level digital reference transactions. 
Rather, it should be viewed as a method of 
providing a measure of control and 
consistency in the assessment process of what 
is a relatively new and rapidly growing 
service role, and which has little in the way of 
precedent or clear and agreed upon 
definitions. 

This is but one example of the difficulty 
in defining terms such that digital reference 
services can be described, measured, 
assessed, and compared. Similar definitional 
problems were encountered in a range of 
other statistics and measures described in this 
manual. Nonetheless, the definitions and 
procedures offered here can provide a starting 
point from which librarians and others can 
begin to evaluate digital reference services on 
an ongoing basis. 
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The Need for Digital Reference 
Evaluation 

In a recent study, Joseph Janes (2002) 
notes that only 9% of survey respondents, 
when queried about digital reference, 
indicated that some type of user evaluation 
had been done as part of providing the 
service. If digital reference services are to 
improve, to better meet user needs, and to 
develop into a successful and high quality 
service, evaluation is essential. Such 
assessment will: 

• Determine the degree to which digital 
reference service objectives are being 
accomplished. 

• Assess how well digital reference services 
support larger library and/or 
organizational goals. 

• Monitor the status of services and 
operations to make ongoing improve- 
ments and modifications with the service. 

• Produce trend data describing use, users, 
and uses of the digital reference service 
over time. 

• Determine the degree to which user 
information needs are being met via the 
service. 

• Justify the costs and benefits of the 
service and demonstrate overall 
accountability of digital reference 
activities. 

• Compare the relative costs and benefits of 
digital reference services to traditional 
reference services. 

• Provide data for ongoing planning and 
development of digital reference services. 

• Determine the types of staff training and 
support that might be necessary. 

• Inform governing boards and other 
organizational administrators of the nature 
and successes of digital reference 
services. 



• Identify those aspects of digital reference 
services that need to be revised or 
improved. 

• Encourage digital reference staff to think 
in terms of outcomes, results, and impacts 
from digital reference services. 

In short, without an ongoing program of 
evaluation and assessment of digital reference 
services, the degree to which this new service 
is “successful” will not be known. 

Purpose of the Manual 

This manual is a first effort to begin to 
identify, describe, and develop procedures for 
assessing various aspects of digital reference 
service. Its overall purpose is to improve the 
quality of digital reference services and assist 
librarians to design and implement better 
digital reference services. More specifically, 
its aim is to: 

• Assist librarians and others to develop 
ongoing programs of assessment of digital 
reference services. 

• Provide a practical set of guidelines and 
procedures for how assessment of digital 
reference services can be done. 

• Develop standardized procedures and 
definitions of statistics and measures that 
may be compared across different 
libraries. 

Ultimately, we hope that this manual will 
assist librarians to better design and develop 
digital reference services in order to better 
meet user information needs. 

As digital reference services develop, 
changes, revisions, and modifications of the 
content of this manual will also be needed. 
We especially expect significant advances to 
take place in the near future in the 
development of software that can 
automatically assist librarians to collect, 
manage, and analyze data describing digital 
reference services. Thus, the authors 
recognize that this manual is a first effort in 
the development of methods, statistics, and 
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measures to assess digital reference. As a first 
effort we expect that others will be able to 
build from this work and extend it to be useful 
as digital reference services continue to 
expand and develop. 

What this Manual is NOT 

This manual is not a literature review of 
resources related to digital reference services. 
There are a number of such reviews available, 
one of which was done by the authors of this 
manual, Assessing Quality in Digital 
Reference Services: Overview of Key 

Literature on Digital Reference (Gross, 
McClure, and Lankes, 2001) and is available 
at : http ://quartz.syr. edu/ quality/VRDphasell 

LitReviw.pdf. Other bibliographies related to 
digital reference services can be found at: 
http://quartz.syr.edu/quality/Reports.htm and 
http://www.lis.uiuc.edu/~b-sloan/digiiref.html 
(Sloan, 2002). 

This manual is neither a practical guide to 
establishing a digital reference service, nor a 
discussion of issues related to the 
development and operation of digital 
reference services. Planning for and 
implementing digital reference services is a 
significant activity in and of itself. This 
manual assumes that the library has the digital 
reference services in place. A beginning 
source for such information has been written 
by Lankes, McClure, Gross, and Pomerantz 
{Implementing Digital Reference Services: 
Setting Standards and Making it Real, 2002). 
Other useful sources can be found in the 
literature reviews discussed above. 

Finally, the manual is not a research 
methods text. The manual assumes that users 
will have a basic knowledge of research 
methods, statistics, data collection techniques, 
etc. Some especially useful research and data 
collection books include: 

• Kruegar, Richard A. and Mary Anne 

Casey. 2000. Focus Groups: A Practical 

Guide to Applied Research. Thousand 

Oaks, CA: Sage Publications. 



• Babbie, Earl. 200 1 . The Practice of Social 
Research. 9th edition. Belmont CA: 
Wadsworth Publishing. 

• Rossi, Peter, Howard E. Freeman, and 
Mark W. Lipsey. 1999. Evaluation: A 
Systematic Approach. 6th edition. 
Thousand Oaks, CA: Sage Publications. 

Clearly, there are numerous other titles 
available. These, however, can provide an 
excellent starting point for users of this 
manual who need additional background 
information in research methods and data 
collection/analysis techniques. 

How to Use this Manual 

The authors recommend various 
assessment methods, statistics, and measures 
throughout the course of this manual, and 
they are summarized in Figure 1.1. For ease 
of use, each statistic and measure has been 
described in terms of the following general 
categories: 

Definition 

Rationale 

Data Collection Procedures 
Issues and Considerations 
For a number of the statistics and measures 
described here, additional detail on rationale, 
issues, and considerations could have been 
included given the situational nature of using 
this manual in different library settings. Since 
all situational considerations cannot be 
covered, the authors provide a general 
depiction and hope that this discussion will be 
of use to those implementing the statistics and 
measures. 

Many of these procedures include data 
collection techniques that have long been 
used to assess services both in the field of 
library and information science and in the 
social sciences in general. These include: 
counting and tabulating of statistics, 
maintenance and analysis of manually kept 
transaction logs, sampling, surveys and 
questionnaires, interviews, and focus groups. 
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FIGURE 1.1 Summary of Measures, Statistics and Standards 

Descriptive Statistics and Measures 

1 . Number of digital reference questions received 

2. Number of digital reference responses 

3. Number of digital reference answers 

4. Number of questions received digitally but not answered or responded to by completely digital 
means 

5. Total reference activity - questions received 

6. Percentage of digital reference questions to total reference questions 

7. Digital reference correct answer fill rate 

8. Digital reference completion time 

9. Number of unanswered digital reference questions 

10. Type of digital reference questions received 

1 1 . Total number of referrals 

12. Saturation rate 

13. Sources used per question 

14. Repeat Users 

Log Analysis 

15. Number of digital reference sessions 

16. Usage of digital reference service by day of the week 

17. Usage of digital reference service by time of day 

18. User’s browser 

19. User’s platform 

User Satisfaction Measures 

20. Awareness of Service 

21. Accessibility of service 

22. Expectations for service 

23. Other sources user tried 

24. Reasons for use 

25. Reasons for non use 

26. Improvements needed/Additional services that need to be offered 

27. Satisfaction with staff service 

28. Delivery mode satisfaction 

29. Impact of service on user 

30. User demographic data 

Cost 

31. Cost of digital reference service 

32. Cost of digital reference service as a percent of total reference budget 

33. Cost of digital reference service as a percent of total library or organizational budget 

Staff Time Expended 

34. Percent of staff time spent overseeing technology 

35. Percent of staff time spent assisting users with technology 

Other Assessment Options 
Peer review 

Enhanced reference transaction logs 
Librarian discussion groups 

Quality Standards 



Statistics, Measures, and Quality Standards for Assessing Digital Reference Library Services: Guidelines and Procedures 



The manual incorporates the use of 
electronically generated logs and reports, 
which are available from software suppliers 
such as Webtrends, Webtracker, and similar 
digital reference software. Examples of these 
digital reports and logs and links to current 
software providers are included in this 
manual. Finally, other assessment options 
may be used, such as peer review, enhanced 
reference logs, and librarian discussion 
groups (see the section titled Other 
Assessment Options). 

Throughout the course of the project 
Assessing Quality in Digital Reference, it 
became clear to the authors that, in many 
cases, digital reference services developed as 
a naturally evolving, “wouldn’t this be great?” 
idea. Little formal planning or gathering of 
input from the potential users took place as 
the services evolved. The authors hope that 
this manual will assist those organizations 
who are thinking about starting or are in the 
initial planning stages of providing some type 
of digital reference (though this manual is not 
a planning manual), as well as those who are 
already providing digital reference. Thinking 
about the evaluation process and the data 
needed to conduct the evaluation prior to 
implementing the service significantly 
improves the evaluation effort. 

Digital reference service is being provided 
in a variety of settings as illustrated above. 
The methods for providing digital reference 
(email, chat, instant messaging, integrated 
voice, interactive video, and specialized 
reference software) in each of these settings 
also vary. For example, email digital 
reference functions in a very different way 
than chat based digital reference software. 

Thus, the statistics and measures, as 
presented here, are more general than specific 
and have been designed to be applied in a 
variety of settings or using a combination of 
methods. This manual does not provide 
detailed individualized assessment measures 
for each of the service provision 



methodologies used. Comments in the Data 
Collection Procedures and Issues and 
Consideration sections give general guidance 
with regard to applications in different digital 
reference contexts. 

Individual libraries or organizations 
providing digital reference may not find all of 
these statistics and measures appropriate for 
their situation. Thus, users should choose 
those assessment statistics and measures that 
will be the most meaningful or relevant for a 
particular situation. Some of the measures 
may be easier to gather or more relevant than 
others, depending on setting and methods 
used. In some libraries it may not be possible 
to gather the data needed for a particular 
measure. In short, the manual is best seen as a 
menu of a range of possible statistics and 
measures from which users can select those 
most relevant and important for their 
particular setting. 

Users of the manual will also recognize 
that some of the statistics have been modified 
from existing statistics of traditional reference 
services. The statistic “number of digital 
reference questions received,” is based on 
traditional approaches. Other traditional 
statistics such as “reference transactions per 
capita” cannot be used in the digital 
environment unless the librarians have an 
accurate assessment of the population they 
intend to serve via the digital reference 
service. Thus, the statistics and measures 
described in the manual blend aspects of 
traditional approaches with new approaches 
that are especially appropriate for a digital 
environment. 

Finally, in using this manual, keep in 
mind that useful evaluation can be done by 
employing only a small number of the 
statistics offered here. Do not attempt to 
immediately start up an evaluation effort and 
try to implement twenty to thirty of these 
statistics and measures. Select specific 
statistics and measures (probably no more 
than three to five on the first iteration) that 
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will be especially relevant and important for 
the first assessment effort. Based on the 
results from this first assessment, then expand 
to collect additional data as appropriate, and 
as your experience and expertise grows. 

Encouraging a Culture of 
Assessment 

Findings from the site visits and field tests 
suggested that ongoing evaluation efforts in 
many libraries are not the norm. Lakos (1999, 
p. 5) defines a culture of assessment as: 

The attitudinal or institutional 
changes that have to occur in order 
for library staff to be able to work in 
an environment where decisions are 
based on facts, research and 
analysis, and services are planned 
and delivered in order to maximize 
positive outcomes and impacts for 
library clients. 

For some library organizations that wish to 
begin an assessment program, significant 
preliminary preparation and planning may be 
needed. 

In the experience of the authors, the 
primary constraints that may affect successful 
ongoing evaluation of services such as digital 
reference include: 

• Administrative support. It is essential that 
top administration and those involved in 
the digital reference process are 
committed to conducting the evaluation, 
willing to commit resources to complete a 
successful evaluation, and will use the 
findings from the evaluation to improve 
the service. 

• The complexity of the service being 
evaluated. For first-time evaluators, the 
authors recommend that they evaluate 
some specific aspect of digital reference, 
i.e., demographics of digital reference 
users, rather than the entire expanse of 
various aspects that constitute digital 
reference. 



• Staff skills and knowledge. Those who 
will be involved in the evaluation effort 
should have basic skills and knowledge in 
research methods, development of data 
collection efforts, statistics and analysis of 
data, and reporting of findings in tables, 
charts, and visually pleasing figures. Staff 
may need training in these areas before 
conducting the evaluation. 

• Organizing staff for evaluation. The 
library staff needs to be organized to 
conduct the evaluation. A person needs to 
be designated as the leader who will 
determine (either alone or with a team) 
how the evaluation should be done, a 
schedule for the evaluation, who will have 
what tasks, the types of statistics and 
measures to be collected, analyzed, and 
reported, etc. 

• Involvement of systems or information 
technology staff. To successfully 
implement a number of the assessment 
techniques (log server analysis, online 
transaction log summaries, etc.) the direct 
involvement of library information 
technology staff is essential. Such a 
person should be part of the evaluation 
team or be actively involved in the 
evaluation effort. 

• Need to organize and manage data. Once 
the data are collected they need to be 
organized, managed, and be made 
available for the long term. Timely reports 
need to be produced and targeted for 
specific user groups and administrators. 
Some type of database or management 
information system should be developed 
to house and manage the data for it to be 
used successfully in the future. 

If these conditions are not present in the 
organization, ongoing evaluation of digital 
reference services will be difficult. Moreover, 
a culture of assessment suggests that library 
staff believe evaluation can improve services, 
that factual data provide an important tool that 
can be used to evaluate services, and that 
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decisions and changes will, in fact, result 
from conducting the evaluation. 

Library staff about to embark on 
evaluation of digital reference services should 
conduct an informal needs assessment to 
determine the degree to which the above 
conditions are present in the library and in 
staff. Based on that needs assessment, some 
initial time to insure that the conditions are 
present is a necessary investment before 
conducting the evaluation. The use of 
consultants to assist in preparing the library 
staff to engage in digital reference assessment 
or in assisting in the assessment itself may be 
appropriate. The success of the evaluation 
effort is as much a factor of the organization’s 
readiness to conduct the evaluation, as is the 
use of the statistics and measures offered in 
this manual. 

Get Over It! 

The fact of the matter is that digital 
reference services will continue to grow and 
develop. To a large degree, the library 
community currently has limited knowledge 
and understanding of the costs, uses, users, 
use, and impacts of digital reference services. 
Evaluation of digital reference services is 
essential if the library community is to better 
understand how to best provide services in the 
networked, remote user environment. A 
critical ingredient to providing “best services” 
is ongoing assessment of digital reference 



services. Simply because the service is 
offered in a digital environment does not 
reduce librarian responsibility to assess that 
service. 

To a large degree, libraries have seriously 
underrepresented their services in terms of 
use of digital services being provided and 
users of the services by not counting and 
assessing these uses and users. As more users 
rely on digital library services - including 
digital reference services - this undercount 
will continue to increase and will continue to 
inadequately describe the extent to which 
digital services are being provided. Thus, 
ongoing ability to describe digital references 
services and include these activities as part of 
overall library services is essential. 

Digital reference and digital reference 
evaluation present a new range of challenges 
for librarians if they wish to prosper in the 
networked environment. When asked how 
librarians can cope with continued changes 
and the other stresses associated with digital 
reference services, McClure was quoted in a 
recent interview (Pierce, 2002, p. 45) as 
stating, “Get Over It.” Digital reference is 
here to stay and the library profession needs 
to learn how to provide these services with 
high quality and high impact, and to be 
assured they are meeting user needs. The 
evaluation procedures outlined in the 
remainder of this manual should provide an 
important initial means for reaching that goal. 
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CHAPTER 2 



DIGITAL REFERENCE STATISTICS, MEASURES AND 

STANDARDS 



This chapter describes the various statistics 
and measures selected for inclusion in this 
manual. The chapter is divided into the 
following sections: 

• Descriptive Statistics And Measures 

• Log Analysis 

• User Satisfaction Measures 

• Cost 

• Staff Time Expended 

• Other Assessment Options 

• Quality Standards 

DESCRIPTIVE STATISTICS 
AND MEASURES 



1. Number of digital reference 
questions received 

Definition: A count of the number of digital 
reference questions received. A digital 
reference question is a question received 
electronically (e.g., via email, webform, 
digital interactive video, digital interactive 
voice, chat or digital reference software which 
incorporates any of the preceding methods) 
and which represents an information need that 
can be characterized as a single concept. To 
be considered a digital reference question, the 
question must be responded to and answered 
using digital means. Do not count any 
questions that may be received digitally but 



that are responded to with non-digital 
methods of communication. These . questions 
should be included in Measure 4 - Number of 
questions received digitally but not answered 
or responded to by completely digital means. 
Do not include any questions that are received 
via phone, voicemail, face-to-face contact, 
fax, and regular postal mail. If more than one 
question is received in a single digital 
transmission (e.g. if a single email message, 
or chat session contains three reference 
questions), each question asked shall be 
counted as a separate question. The 
transmission itself shall not count as an 
additional question. 

Rationale: This statistic is important for 
determining the volume of service. The 
numbers derived from this measure can be 
compared with the number of digital 
reference responses and digital reference 
answers to determine output. This statistic can 
also be compared to the total number of 
reference questions (of any type) received by 
a library or organization offering both 
traditional and digital reference service and 
can assist in the determination of appropriate 
staffing needs. A comparison of the varying 
types of digital reference services that might 
be provided by a single library or digital 
reference service (i.e., chat and email) could 
also be derived from this statistic. This 
statistic can assist in decisions regarding 
promotion of the service. 
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reference responses and Measure 3 — 
Number of digital reference answers. 

Do not count as Digital Reference 
Questions Received any questions that are 
received digitally but which are responded 
to with non-digital methods of 
communication. These questions should 
be included in Measure 4 - Number of 
questions received digitally but not 
answered or responded to by completely 
digital means. 

This measure should be used in 
conjunction with the following measures: 

2 - Number of digital reference responses. 

3 — Number of digital reference answers. 

4 - Number of questions received digitally 

but not answered or responded to by 
completely digital means. 

5 — Total reference activity - questions 

received. 

6 - Percentage of digital reference 

questions to total reference questions. 



Data Collection Procedure: Continuous 
collection or Sampling. 

Collected by: Staff (using manual or 
electronically generated log or report or 
combination of two depending on 
situation). 

Frequency: Logs are kept daily. Data is 

collected and tabulated in two-week 
periods and is generally reported annually 
unless required more frequently. 
Procedures: Instruct staff on the 

definition of this measure as defined 
above. Inform staff of any special 
considerations they should take into 
account. 

Continuous collection: Keep a record 
of the number of digital reference 
questions received daily, collect and 
tabulate every two weeks. Reporting is 
most frequently done on an annual 
basis. 

Sampling: Select a typical two-week 
period. Keep a record of each digital 
reference question received on a daily 
basis (if using a manual log), or count 
the total number of reference 
questions received over the two-week 
period if using an electronic log. 
Manual analysis of electronic logs 
(generated for email, chat sessions, 
etc.) will be required to account for 
transmissions that may contain more 
than one digital reference question. 
Once data for the two- week period has 
been tabulated, extrapolate for the 
entire year based on the sample data. 
Collect data for sample two-week 
periods in various months throughout 
the year to account for periods of 
expected peak or diminished use. 

For this measure count only those 
questions received digitally that are also 
responded to and answered digitally as 
defined in Measure 2 — Number of digital 



Helpful tools - See Appendix I for: 

H. Sample Digital Reference Transaction 
Record. 

I. Digital Reference Question 
Transaction Log - Email. 

J. Digital Reference Question 
Transaction Log - Real Time 
Sessions. 

K. Total Reference Activity Log - Daily / 
Two Week Period. 

M. Sample Digital Reference Data 

Collection Worksheet — Compilation. 

Issues and Considerations: Depending on 
volume it can be time consuming to keep an 
accurate and detailed record of the number of 
reference questions received on a daily basis. 
Many libraries keep a manual tally sheet of 
the number of reference questions received 
(both traditional and digital), while other 
organizations will keep a sample log over the 



Statistics, Measures, and Quality Standards for Assessing Digital Reference Library Services: Guidelines and Procedures 



course of a brief period of time (such as a 
typical week or two at various times 
throughout the year) and then will project the 
count for the entire year. Some libraries 
utilize digital reference tracking software, 
while others use a combination of both 
manual and electronic logs. For those libraries 
offering digital reference using email or chat 
software, special attention and training should 
be given in recognizing the number of distinct 
reference questions received during a single 
session. Aggregating the numbers from these 
various sources (especially in situations where 
a library may be providing separate digital 
and traditional reference service) may require 
coordination of several staff members with 
attention given to consistency of counting 
methods and avoiding duplication of counts 
or omission of counts. Also of consideration 
when doing sampling is to account for periods 
of expected peak and diminished activity 
(such as summer and before finals in an 
academic setting, and holidays and summer 
for public library settings). 

2. Number of digital reference 
responses 

Definition: The number of digital responses 
refers to the number of digital contacts that 
are made with the user by staff in order to 
address a digital reference question as defined 
in Measure 1 - Number of Digital Reference 
Questions Received. Do not count as 
responses digital contacts initiated by the 
user. A digital response refers to any 
communication from the staff member to the 
user that is transmitted digitally. These digital 
contacts may occur via email, webform, 
interactive video sessions, interactive voice, 
chat or reference software. If more than one 
email message is sent to the user or more than 
one chat or real-time interactive video or 
voice session occurs in response to a digital 
reference question, each of these is counted as 
a separate response. Do not count the 



individual messages back and forth during a 
singe chat session as separate responses; each 
chat session counts as a single response. 
There may be more than one response to a 
single digital reference question. It should be 
noted that a digital response is not the same as 
a digital answer, i.e., a digital response to the 
user may or may not include the answer. A 
digital response may be a request for more 
information from the user, or may be a 
message asking if the digital answer sent was 
what the user expected. Do not include any 
responses that are sent to the user via phone, 
voicemail, face-to-face contact, fax, or regular 
postal mail. 

Rationale: This fundamental statistic will 
indicate how many responses are sent in the 
process of trying to answer a digital reference 
question as defined above in Measure 1 - 
Number of Digital Reference Questions 
Received. This statistic can be used to analyze 
such factors as possible breakdowns in 
technology (effectiveness measure), 
effectiveness of the digital reference interview 
process, or the level of effort being expended 
in answering the question (efficiency 
measure). 

Data Collection Procedure: Continuous 
collection or Sampling. 

Collected by: Staff (using manual or 
electronically generated log or report or 
combination of two depending on 
situation). 

Frequency: Logs are kept daily. Data is 

collected and tabulated in two-week 
periods and is generally reported annually 
unless required more frequently. 
Procedures: Instruct staff on the 

definition of this measure. Inform staff of 
any special considerations they should 
take into account. 

Continuous collection: Keep a record 
of the number of digital responses 
made to each digital reference 
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question received daily, collect and 
tabulate every two weeks. Reporting is 
most frequently done on an annual 
basis. 

Sampling: Select a typical two-week 
period. Keep a record of the number 
of digital responses made to each 
digital reference question received on 
a daily basis (if using a manual log), 
or count the total number of responses 
made to each digital reference 
question received over the two-week 
period if using an electronic log. Some 
manual analysis of electronic logs 
(generated for email, chat sessions, 
etc.) will be required to determine the 
number of responses made to one 
question. Once data for the two-week 
period has been tabulated, extrapolate 
for the entire year based on the sample 
data. Collect data for sample two- 
week periods in various months 
throughout the year to account for 
periods of expected peak or 
diminished use. 

This measure should be used in 

conjunction with the following measures: 

1 — Number of digital reference questions 

received. 

3 — Number of digital reference answers. 

4 — Number of questions received digitally 

but not answered or responded to by 
completely digital means. 

5 - Total reference activity - questions 

received. 

6 - Percentage of digital reference 

questions to total reference questions. 

Helpful tools - See Appendix I for: 

H. Sample Digital Reference Transaction 
Record. 

I. Digital Reference Question 
Transaction Log - Email. 



J. Digital Reference Question 
Transaction Log - Real Time 
Sessions. 

K. Total Reference Activity Log - Daily / 
Two Week Period. 

M. Sample Digital Reference Data 
Collection Worksheet - Compilation. 

Issues and Considerations: Depending on 
volume it can be time consuming to keep an 
accurate and detailed record of the number of 
digital reference responses provided on a 
daily basis. Recording this statistic will 
require that, after each question is answered, 
the staff member record whether the question 
was responded to or answered entirely 
through electronic means, as defined above, 
or if the responses or answer were provided 
via traditional means or a via a combination 
of traditional and digital responses. Another 
area for consideration is if more than one staff 
member contributes to the answering of a 
question. Additionally, some libraries only 
keep a tally sheet of questions received and 
not of questions answered (either through 
traditional or digital means) or of responses 
made because they have previously not been 
required to keep track of the these statistics. 
This new procedure of detailed tracking of the 
number of digital responses will require 
additional training and understanding of the 
concepts involved. It will also involve 
additional time. 

3. Number of digital reference 
answers 

Definition: A digital reference answer 

provides the information that the user is 
seeking for their digital reference question. 
For the purposes of this manual, the aggregate 
of electronic responses made to a single 
digital reference question as defined in 
Measure 1 — Number of Digital Reference 
Questions Received shall count as single 
digital reference answer. This aggregate must 
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include at least one of the following: a) 
communication with the user that provides an 
actual specific answer to the reference 
question (i.e., actual content related to the 
subject of the question); b) directions to 
resources that may help the patron locate the 
answer on their own (as often occurs in digital 
reference); or c) a referral to another contact 
point, either external or internal, as defined in 
Measure 11 — Total Number of Referrals. Do 
not count as a digital reference answer any 
aggregate of responses that does not provide 
content that directly addresses the user’s 
information need, does not provide specific 
directions to the user about resources that may 
help them locate the answer on their own, or 
is not a referral. Do not count as a digital 
reference answer the aggregation of a series 
of responses that ask the user for more 
information yet are never responded to (user 
evaporation) or a series of responses that 
result in the user not receiving the information 
they need. Do not count as a Digital 
Reference Answer any answer that is 
provided to the user digitally, but which may 
have been preceded by a non-digital response. 
(For example, the question may have been 
received digitally and the answer provided 
digitally, but the staff member may have 
called the user for more information.). Again, 
for any digital reference transaction to be 
counted as digital, each component of the 
transaction (i.e., the question, responses, and 
answer) must be transmitted digitally. 

Rationale: A digital reference answer as 
defined above will be useful in determining 
how many digital reference questions 
received actually have an answer provided to 
the user. For the purposes of this manual an 
answer can also include a referral. This 
statistic will be useful when completing other 
measures such as Measure 8 - Digital 
Reference Completion Time, and the User 
Satisfaction Measures, which relate to users’ 
perceptions regarding various aspects of their 



answer. This measure can assist in collection 
and staff development decisions if the ratio of 
answers to questions is low. 

Data Collection Procedure: Continuous 
collection or Sampling. 

Collected by: Staff (using manual or 
electronically generated log or report or 
combination of two depending on 
situation). 

Frequency: Logs are kept daily. Data is 

collected and tabulated in two-week 
periods and is generally reported annually 
unless required more frequently. 
Procedures: Instruct staff on the 

definition of this measure. Inform staff of 
any special considerations they should 
take into account. 

Continuous collection: Keep a record 
of the number of digital reference 
answers provided daily, collect and 
tabulate every two weeks. Reporting is 
most frequently done on an annual 
basis. 

Sampling: Select a typical two-week 
period. Keep a record of the number 
of digital reference answers provided 
on a daily basis (if using a manual 
log), or count the total number of 
answers made to each digital reference 
question received over the two-week 
period if using an electronic log. Some 
manual analysis of electronic logs 
(generated for email, chat sessions, 
etc.) will be required to determine the 
number of digital reference answers. 
Once data for the two-week period has 
been tabulated, extrapolate for the 
entire year based on the sample data. 
Collect data for sample two-week 
periods in various months throughout 
the year to account for periods of 
expected peak or diminished use. 

Again, count the collection of these 
responses as one answer. Do not count as 



Statistics, Measures, and Quality Standards for Assessing Digital Reference Library Services: Guidelines and Procedures 



a digital reference answer any responses 
sent to the user asking for more 
information if such a request is never 
responded to by the user (these will be 
counted in Measure 9 - Number of 
Unanswered Digital Reference 
Questions). Also, do not count as a digital 
reference answer any aggregation of 
responses that include any form of 
communication with the user that is not 
considered digital. For example, if during 
the course of the reference transaction a 
telephone or face-to-face conversation 
takes place, the answer will not be 
considered a digital reference answer even 
if the question was submitted digitally. 

This measure should be used in 
conjunction with the following measures: 

1 — Number of digital reference questions 

received. 

2 - Number of digital reference responses. 

4 — Number of questions received digitally 

but not answered or responded to by 
completely digital means. 

5 - Total reference activity — questions 
received. 

6 - Percentage of digital reference 

questions to total reference questions. 

Helpful Tools - See Appendix I for: 

H. Sample Digital Reference Transaction 
Record. 

I. Digital Reference Question 
Transaction Log - Email. 

J. Digital Reference Question 
Transaction Log - Real Time 
Sessions. 

K. Total Reference Activity Log - Daily / 
Two Week Period. 

M. Sample Digital Reference Data 
Collection Worksheet — Compilation. 

Issues and Considerations: A review of the 
literature and research that has sought to 
arrive at a consistent and meaningful 



definition of what constitutes an “answer” to a 
reference question (either digital or 
traditional) has shown that such a definition is 
elusive. Because of the difficulty in defining 
what exactly is meant by the term “answer,” 
and because the patron’s definition of answer 
may not match the definition given here, 
collection of this measure will require careful 
scanning of the responses made to the user 
and thoughtful analysis. This will be 
especially so for the more complex reference 
questions. When doing this analysis, such 
questions as “Do we count partial answers?” 
and “How do we know when the answer is 
complete?” may arise. Consistency in how 
these questions are answered will be 
important in collecting the statistics for this 
measure. Providing an answer to a user’s 
digital reference question is the primary 
reason that a reference service exists, and 
while this statistic may be more difficult to 
collect it is also one of the most important. 

4. Number of questions received 
digitally but not answered or 
responded to by completely 
digital means. 

Definition: This measure refers to those 
reference questions that are received digitally 
but have been responded to and/or answered 
either with traditional methods or using a 
combination of both digital and non-digital 
communication methods. NOTE: For the 
purposes of this manual, the number gathered 
for this measure shall not be included in the 
count of Measure 1 - Digital Reference 
Questions Received but shall be counted here 
as a separate measure. 

Rationale: In many cases, questions that are 
received digitally are responded to using a 
combination of methods, some digital and 
some not. This is especially true of those 
libraries that provide a combination of both 
traditional and digital reference service. For 
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example, a question received via email may 
warrant a response that includes a phone call 
to the user for more information, sending a 
book to the user via a branch library, or 
faxing a required article in addition to sending 
an article electronically. In many cases, these 
types of transactions comprise a significant 
portion of the transactions that take place in a 
library. This measure, while not meeting the 
strict criteria as a digital reference question as 
defined in this manual (i.e. must be responded 
to and answered by completely digital 
means), will allow for an additional point of 
comparison between digital reference and 
traditional reference and will also allow for 
recognition of what, in some cases, may be a 
significant portion of reference activity. This 
will give a better overall picture in situations 
where both traditional and digital reference 
service are provided. This measure will be 
useful in making staffing, training, and 
collection development and operating expense 
decisions. For example, if there is a large 
number of these type of questions, staff will 
need to be familiar with both traditional and 
digital resources, and will need to make 
decisions about the best resource to answer 
the question and the best way to respond to 
the patron. Staff may need training in how to 
use both traditional and digital resources. 
Collection development decisions can include 
whether to buy print or digital resources or 
both. Allocation of operating expenses can 
include deciding whether additional voice 
phone lines, fax machines, and workstations 
for email service are needed. Libraries 
offering reference service that simultaneously 
provides both digital and traditional methods 
must necessarily look closely at the reference 
service as a whole, and this measure will help 
provide a clearer picture of the relative 
proportions of the various aspects of the 
service that they provide. 



Data Collection Procedure: Continuous 
collection or Sampling. 

Collected by: Staff (using manual or 
electronically generated log or report or 
combination of two depending on 
situation). 

Frequency: Logs are kept daily. Data is 

collected and tabulated in two-week 
periods and is generally reported annually 
unless required more frequently. 
Procedures: Instruct staff on the 

definition of this measure and correct 
procedures. Inform staff of any special 
considerations they should take into 
account. 

Continuous collection: Keep a record 
of the number of digital reference 
questions received but not answered or 
responded to by digital means daily. 
Collect and tabulate every two weeks. 
Reporting is most frequently done on 
an annual basis. 

Sampling: Select a typical two-week 
period. Keep a record of the number 
of digital reference questions received 
but not answered or responded to by 
digital means on a daily basis. 
Analysis of manual and electronic logs 
(generated for email, chat sessions, 
etc.) will be required. Once data for 
the two-week periods has been 
tabulated, extrapolate for the entire 
year based on the sample data. Collect 
data for sample two-week periods in 
various months throughout the year to 
account for periods of expected peak 
or diminished use. 

This measure should be used in 
conjunction with the following measures: 

1 - Number of digital reference questions 

received. 

2 - Number of digital reference responses. 

3 - Number of digital reference answers. 

5 - Total reference activity - questions 

received. 
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6 — Percentage of digital reference 

questions to total reference questions. 

Helpful Tools - See Appendix I for: 

H. Sample Digital Reference Transaction 
Record. 

I. Digital Reference Question 
Transaction Log - Email. 

J. Digital Reference Question 
Transaction Log - Real Time 
Sessions. 

K. Total Reference Activity Log - Daily / 
Two Week Period. 

M. Sample Digital Reference Data 

Collection Worksheet - Compilation. 

Issues and Considerations: It is 

acknowledged by the authors that in many 
situations questions received digitally may not 
be responded to and/or answered digitally. 
For example, a question may be sent via 
email, but the staff member may call the 
patron on the phone for clarification or may 
need to fax an article to the patron. This 
measure addresses and gives credit for those 
hybrid-type reference transactions and will be 
helpful for those libraries or services that 
provide a combination of reference services. 
It may be less useful to those digital reference 
services that are strictly designated as such 
and are not part of a larger library service. 

5. Total reference activity - 
questions received 

Definition: Total reference activity refers to 
the total number of reference questions 
received and will be calculated by adding the 
number of digital reference questions received 
( Measure 1 — Digital Reference Questions 
Received), the number of questions received 
digitally but not responded to with completely 
digital means {Measure 4 — Number of 
questions received digitally but not answered 
or responded to by completely digital means) 



to reflect hybrid transactions, and the number 
of traditional reference questions received. 



DQR + HQR = TQR 



Rationale: Libraries are increasingly 

providing both separate traditional and digital 
reference services or in many cases a 
combination of both, and it is useful to know 
what the total reference activity is as defined 
by the total number of reference questions 
received. This statistic will be useful in 
assessing the total reference service provided 
to users in the context of all library services 
and can be beneficial in long range planning. 

Data Collection Procedure: Continuous 
collection or Sampling. 

Collected by: Staff (using manual or 
electronically generated log or report or 
combination of two depending on 
situation). 

Frequency: Logs are kept daily. Data is 

collected and tabulated in two-week 
periods and is generally reported annually 
unless required more frequently. 
Procedures: Instruct staff on the 

definition of this measure as defined 
above. Inform staff of any special 
considerations they should take into 
account. 

Continuous collection: Keep a record 
of the total number of reference 
questions received (traditional, hybrid 
and digital) daily, collect and tabulate 
every two weeks. Reporting is most 
frequently done on an annual basis. 
Sampling: Select a typical two-week 
period. Keep a record of each 
reference question received 
(traditional, hybrid and digital) on a 
daily basis using both manual and 
electronic logs. Some manual analysis 
of electronic logs (generated for email, 
chat sessions, etc.) will be required to 
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account for transmissions that may 
contain more than one digital 
reference question. Once data for the 
two-week period has been tabulated, 
extrapolate for the entire year based 
on the sample data. Collect data for 
sample two-week periods in various 
months throughout the year to account 
for periods of expected peak or 
diminished use. 

Add the number of digital reference 
questions received, the number of 
reference questions received which are not 
responded to entirely through digital 
means, and the number of traditional 
reference questions received. 

In some cases, digital reference staff may 
need to coordinate with the staff providing 
traditional or integrated reference service 
if the services are separate in the library. 

If sampling is the chosen method of 
collection for this measure, coordination 
will also be required so that sampling 
periods for traditional or integrated 
reference service coincides with the 
sampling periods for the digital reference 
service. 

This measure should be used in 
conjunction with the following measures: 

1 - Number of digital reference questions 
received. 

2 - Number of digital reference responses. 

3 - Number of digital reference answers. 

4 - Number of questions received digitally 

but not answered or responded to by 
completely digital means. 

6 - Percentage of digital reference 

questions to total reference questions. 

Helpful Tools - See Appendix I for: 

I. Digital Reference Question 
Transaction Log - Email. 



J. Digital Reference Question 
Transaction Log — Real Time 
Sessions. 

K. Total Reference Activity Log - Daily / 
Two Week Period. 

M. Sample Digital Reference Data 

Collection Worksheet - Compilation. 

Issues and Considerations: This measure will 
be of more use to those libraries providing a 
combination of reference services (i.e., both 
traditional and digital). Keeping track of the 
number of reference questions received by 
either traditional or digital means on a daily 
basis is time consuming. Determining which 
category each reference transaction falls into 
will require manual analysis of each 
transaction. Many libraries utilize a sampling 
methodology where they keep a sample log 
over the course of a brief period of time (such 
as a typical week or two at various times 
throughout the year) and then project the 
count for the entire year. Other libraries 
utilize digital reference tracking software. 
Some use a combination of both manual and 
electronic logs. Aggregating the numbers 
from these various sources may require 
coordination of several staff members with 
attention required to consistency of counting 
methods and avoiding duplication of counts. 
Also of consideration when doing sampling is 
to account for periods of peak and lesser 
activity (such as summer and before finals in 
an academic setting, and holidays and 
summer for public library settings). 

6. Percentage of digital reference 
questions to total reference 
questions. 

Definition: A percentage of the number of 
digital reference questions received to total 
number of reference questions received. The 
digital reference question must be received 
electronically (e.g., via email, WWW form, 
interactive video, interactive voice, chat, 
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reference software) and answered and 
responded to digitally as defined in Measure 1 
- Number of digital reference questions 
received. The number of total reference 
questions received is obtained in Measure 5 - 
Total Reference Activity - Questions 
Received. 

Digital Reference Questions Received X 100 
Total Reference Questions Received 



Rationale: Because most libraries operate on 
limited funding and must allocate their 
resources carefully, it is critical for sound 
library management that the percentage of 
digital reference questions received be 
compared to the total number of reference 
questions received. This percentage can then 
be used in making both short term 
management and long range planning 
decisions. For example, if 80% of a library’s 
reference staff is dedicated to the provision of 
traditional reference service while 70% of the 
reference questions received are received 
digitally, serious consideration must be given 
to why this is so and how the situation should 
be managed. Does more staff need to be 
devoted to the provision of digital reference 
service if the service is separate from the 
traditional service? In cases where the same 
staff members provide both digital and 
traditional reference, does more staff training 
need to be provided? Do more electronic 
resources need to be purchased? Conversely, 
if a library is committed to the provision of 
digital reference service and allocates a large 
percentage of the budget to the provision of 
digital reference yet only 15% of the 
questions received are received digitally, an 
analysis must be made to determine if the 
digital reference service is something the user 
population needs at the level to which it has 
been funded or if the service has been 
advertised adequately. 



0 




Data Collection Procedure: Continuous 
collection or Sampling. 

Collected by: Staff (using manual or % 

electronically generated log or report or 
combination of two depending on 
situation). 

Frequency: Reported annually, though 

logs are kept on a daily basis and 4 

collected and tabulated on a per week 
basis. 

Procedures: Instruct staff on the 

definition of this measure. Inform staff of 
any special considerations they should ^ 

take into account. 

Continuous collection: Keep a record 
of reference questions received daily, 
tabulate every two weeks. Reporting is 
most frequently done on an annual 
basis. " 

Sampling: Select a typical two-week 
period, keep a record of all digital 
reference questions received on a daily 
basis, then extrapolate based on the 
sample data. Collect data for sample • 

two-week periods in various months 
throughout the year to account for 
periods of peak or diminished use. 

Divide the total number of digital 4 

reference questions received by the total 
number of reference questions received 
and then multiply by 1 00 . 

For this measure the digital reference staff ^ 

will need to coordinate with the staff 
providing traditional reference service if 
the services are separate in the 
organization. 

Coordination will also be required so that ® 

sampling periods for traditional reference 
service coincide with the sampling periods 
for the digital reference service if 
sampling is the chosen method of 
collection for this measure. ® 
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This measure should be used in 

conjunction with the following measures: 

1 - Number of digital reference questions 

received. 

2 - Number of digital reference responses. 

3 - Number of digital reference answers. 

4 - Number of questions received digitally 

but not answered or responded to by 
completely digital means. 

5 - Total reference activity — questions 
received. 

Helpful Tools - See Appendix I for: 

I. Digital Reference Question 
Transaction Log — Email. 

J. Digital Reference Question 
Transaction Log — Real Time 
Sessions. 

K. Total Reference Activity Log - Daily / 
Two Week Period. 

M. Sample Digital Reference Data 

Collection Worksheet - Compilation. 

Issues and Considerations: This measure will 
be most useful for those libraries providing a 
combination of reference services (i.e., digital 
and traditional). Again, keeping track of the 
number reference questions received on a 
daily basis can be time consuming. One 
method for measuring the number of 
reference questions received is to pick a short 
time period (a week or two), which represents 
a typical period of use for the service, to 
accurately measure for that time period and to 
then project the count over the course of the 
year. Peak periods and periods of less use 
should also be taken into account when 
projecting the count. Another consideration 
with this measure is in carefully determining 
how the data will best be used. 



7. Digital reference correct answer 
fill rate 

Definition: Digital reference correct answer 
fill rate is the ratio of correct answers given to 
total answers given. 

Correct Digital Reference Answers X 100 
Total of Digital Reference Answers 



Rationale: With any library service it is 
important to determine if the service is 
accomplishing its stated goals. In the 
provision of any reference service (either 
traditional or digital) one of the most 
fundamental and logical goals is that correct 
answers be given to the reference questions 
received. Indeed this measure should be one 
of the keystones in assessing the quality of 
digital reference. A review of library literature 
on the provision of correct answers to 
reference questions has shown that in many 
cases the provision of correct answers to 
reference questions occurs less frequently 
than would be expected of a quality reference 
service. Other literature has shown that 
correctly answering reference questions using 
the Internet is equal to or exceeds the 
provision of correct answers to reference 
questions using traditional resources. The 
information gathered from this measure can 
be crucial in making collection development 
decisions (Are the resources being used 
outdated or not updated frequently enough? 
Are the resources needed to answer such 
questions available to staff? Are the resources 
reputable?), negotiating contracts with 
database vendors, and developing continuing 
education programs for staff members. 



Data Collection Procedure: 

Peer Review. 

Unobtrusive testing. 

Independent analysis of answers by outside 
observer. 
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Collected by: Management or supervisor. 
Frequency: Annually (at least 

minimally) or more frequently if 
necessary or useful. 

Procedures: 

Peer review: Periodically, staff 

members review case studies of actual 
reference transactions that occurred 
over a defined period of time, (the 
previous two weeks for example), in 
which answers were provided to 
digital reference questions. All 
identifying information must be 
removed regarding both the user and 
the staff members who handled the 
question. The goal is to assess whether 
the question was answered in the best 
way possible and to share suggestions 
for improvements or additional 
resources that could help to answer 
similar questions in the future. The 
peer review group reaches consensus 
regarding the “correctness” of the 
answer provided. This method is 
useful in assessing quality and 
correctness in those questions that 
may be more complex than the simple 
ready reference question (though this 
method could also be used for those 
types of questions). 

Unobtrusive testing: A set of quick 
fact ready reference questions with 
predetermined known answers are 
submitted to the digital reference 
service without the staff members 
knowing that these questions are part 
of the assessment process. A 
supervisor, manager or outside 
assessor reviews the answers received. 
An analysis of the answers given is 
used to calculate the correct answer 
fill rate. 

Independent analysis of answers by 
outside observer: Periodically an 

outside analyst or team of analysts 
reviews answers provided to digital 



reference questions over a given 
period of time. Identifying information 
regarding staff members who handled 
the questions is removed before the 
analysis period. The outside analysts 
assess whether each answer given was 
“correct” or “incorrect” and the 
correct answer fill rate is calculated. 
The data is then given to the manager 
or supervisor. 

Helpful Tools — See Appendix I for: 

H. Sample Digital Reference Transaction 
Record. 

M. Sample Digital Reference Data 

Collection Worksheet - Compilation. 

Issues and Considerations: When a user 
submits a question to a digital reference 
service he or she is expecting to receive a 
correct answer. In fact, the provision of a 
correct answer is one of the most important 
determinants in assessing quality in digital 
reference (or any other type of reference 
service). For many ready reference type 
questions (which, in many cases, are the only 
type of questions answered by a digital 
reference service) it is expected that giving 
the correct answer would be relatively easy 
and that the correct answer fill rate would be 
high. However, decades of research have 
shown that this is not necessarily the case. 
Determining whether an answer is “correct” 
or not becomes even more difficult for more 
complex types of questions (such as in-depth 
research questions, literature reviews, etc.). 

Staff and management alike may be resistant 
to the idea of being tested on the correctness 
of the answers they provide to their users and 
there may be union restrictions regarding such 
testing. However, such testing, if done in a 
sensitive, constructive, and non-threatening 
manner can be extremely useful for a number 
of reasons. The data gathered from this 
measure can bolster staff members’ 
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contentions that newer resources are needed 
or that they would like training in a particular 
subject area. 

Sharing of knowledge through the peer 
review process can provide an opportunity for 
displaying areas of expertise or interest that 
staff members may have. Unobtrusive testing, 
while admittedly limited in the types of 
questions and answers it can address, provides 
for the most realistic assessment of “correct” 
answers. 

It is the authors’ firm belief that, though there 
may be resistance from staff and managers 
alike, determining the correct answer fill rate 
is an essential measure for assessing quality in 
the provision of digital reference. 

8. Digital reference completion 
time 

Definition: Digital reference completion time 
refers to the average time that it takes to 
provide a digital reference answer to a digital 
reference question. This measure is often 
referred to as turnaround time. 

Rationale: How long it takes to provide an 
answer to a digital reference question is a 
crucial factor when assessing the quality of 
any reference service. This measure can be 
used to determine if answers are being 
provided in a timely manner and can also be 
used to set policy for expected turnaround 
time for the service based on the resources 
available. 

Data Collection Procedure: Sampling. 

Collected by: Staff (using manual or 
electronically generated log or report or 
combination of the two depending on 
situation). 

Frequency: Logs are kept daily. Data is 

collected and tabulated in two-week 



periods and is generally reported annually 
unless required more frequently. 
Procedures: Instruct staff on the 

definition of this measure. Inform staff of 
any special considerations they should 
take into account. 

Sampling: Select a typical two-week 
period, keep a count on a daily basis, 
collect and tabulate for selected 
sample two-week period, and then 
extrapolate based on the sample data. 
Collect data for sample two-week 
periods in various months throughout 
the year to account for periods of peak 
or diminished use. 

Count only those questions that are both 
initiated and completed in the same two- 
week period. 

On the reference log, staff should note the 
date and time that the digital reference 
question was sent by the user (for email 
and web submission form) or the date and 
time the initial contact was initiated by the 
user of real time services (via chat, 
interactive voice or video, or digital 
reference software). The log should also 
note the date and time that the initial 
response was made by the service staff to 
the user, and the date and time that the 
final response was sent to the user. Only 
count the time if an answer or referral was 
actually provided to the user. 

The units of measurement should be 
counted in minutes, hours and days (or 
any combination) depending on the type 
of service that is provided and the needs 
of the particular service. The time that has 
transpired for each transaction is 
measured and at the end of the sample 
two-week period the average is calculated. 
Include all time that transpires between 
the submission of the question and the 
final response. 
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Over the course of the two-week sample 
period there may be one or two 
transactions (either real time or chat) that 
take much longer than others and thus 
may not present a realistic view of the 
average. An analysis of the median and 
distribution in such cases may give a more 
accurate picture of the completion time or 
turnaround time for die service. 

NOTE: Do not subtract time that the staff 
members may not have been working on 
the question, working on other questions 
or on break. All time that has elapsed 
should be counted regardless of the time 
that was actually spent on the specific 
question. 

Helpful Tools - See Appendix I for: 

H. Sample Digital Reference Transaction 
Record. 

I. Digital Reference Question 
Transaction Log - Email. 

J. Digital Reference Question 
Transaction Log - Real Time 
Sessions. 

L. Digital Reference Completion Time 
Calculation Worksheet. 

M. Sample Digital Reference Data 
Collection Worksheet - Compilation. 

Issues and Considerations: Again, it may be 
time consuming and difficult to provide the 
level of detailed record keeping this measure 
requires on a daily continuous basis and it is 
recommended that a typical sample period 
(two weeks) be measured. Some may feel that 
this measure does not assess the actual time 
spent working on the question. In many cases, 
staff members may be interrupted, may be 
working on more than one question at a time, 
may take breaks or lunch or even go home for 
the night before completing a question. Also, 
more than one staff member may be working 
on the same question. The authors, advisory 



committee, and field testers felt that it might 
be difficult for some libraries to record every 
time staff members started and stopped 
working on question at such a detailed level. 
If a library or service has the capacity and 
resources to provide this level of detail they 
are encouraged to do so as a separate 
measure, but it should be noted that the 
primary purpose of this measure is to 
determine from the user’s perspective how 
long it took for his or her question to be 
answered. It should not be assumed that a 
quick turnaround time is always the most 
desirable outcome. In situations where the 
question is complex or the user expects a 
comprehensive answer, a longer turnaround 
time may be a better indicator of quality than 
a shorter one. Also, strict emphasis on 
turnaround time may cause the staff to 
sacrifice quality reference interviews and 
appropriate follow-up and may also 
encourage inappropriate referrals. This 
measure should be closely looked at and 
analyzed with sensitivity and thought. 

9. Number of unanswered digital 
reference questions 

Definition: An unanswered question is a 
digital reference question for which no 
answer (as defined in Measure 3 — Number of 
Digital Reference Answers) is sent by the 
service to the user. Include in this count any 
“incomplete” reference transactions. An 
incomplete digital reference transaction is 
defined as a reference transaction in which a 
request for further information sent by the 
staff to the user in response to digital 
reference question results in no further 
communication on the part of the user. This 
phenomenon is referred to in the literature as 
“user evaporation” (January 18, 2002 coined 
by Joe Janes at ALA Mid-Winter 2002 
Conference in New Orleans for the ACRL 
Program Digital Reference: Trends, 

Techniques and Change). Also include any 
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questions that have not been answered 
because of capacity issues (i.e., not enough 
staff available to answer the question in the 
required time or not enough resources). 
Finally include any questions that may have 
no answers, or any questions that may have 
“slipped through the cracks” due to technical 
or human error (for example a question was 
sent but no response whatsoever was made by 
the staff to the users). Do not include in this 
count any questions for which a referral 
(either internal or external as defined in 
Measure 11 — Total Number of Referrals) is 
made. A referral is counted as a digital 
reference answer as defined in Measure 3 — 
Number of Digital Reference Answers above. 

Rationale: A count of unanswered digital 
reference questions can help determine if the 
capacity of the service is adequate. This 
statistic can also assist in assessing the quality 
of the communication skills of the staff. An 
analysis of the unanswered reference 
questions can determine if the number of 
technical problems or human errors is 
excessive, or if expectations of the users are 
realistic or unrealistic (as in the case of 
unanswerable questions). In addition, this 
count can be compared to the total number of 
digital reference questions received and/or the 
total number of digital reference questions 
answered to derive ratios such as: 
“Unanswered digital reference questions” to 
“Total reference questions” 

“Digital reference questions” to “Digital 
reference answers.” 

Data Collection Procedure: Continuous 
collection or Sampling. 

Collected by: Staff, manager or 
supervisor. 

Frequency: Logs are kept daily. Data is 

collected and tabulated in two-week 
periods and is generally reported annually 
unless required more frequently. 



Procedures: Inform staff of the correct 

definition of this measure. A designated 
staff person, manager, or supervisor 
collects and reviews the logs after a given 
period of time and analyzes them to 
determine if there are any unanswered 
questions as defined above. 

Continuous collection: Logs are kept 
daily. Data is collected and tabulated 
every two weeks and is generally 
reported annually unless required 
more frequently. 

Sampling: Select a typical two week 
period, keep a count on a daily basis, 
collect and tabulate for selected 
sample period and then extrapolate 
based on the sample data. Collect data 
for sample two-week periods in 
various months throughout the year to 
account for periods of peak or 
diminished use. 

Helpful Tools - See Appendix I for: 

H. Sample Digital Reference Transaction 
Record. 

M. Sample Digital Reference Data 

Collection Worksheet - Compilation. 

Issues and Considerations: Determining the 
number of unanswered digital reference 
questions will require keeping a log (either 
manual or with digital reference software) and 
a periodic analysis of this log by supervisors 
or staff or a peer review group. Another issue 
is when to decide that a question is 
unanswered (especially if the unanswered 
question is the result of an incomplete 
reference transaction). Libraries may set a 
specific time after which a question is 
designated unanswered. For example, if no 
answer is provided within 3 or 4 days the 
question may be considered unanswered. This 
designated time period may depend largely on 
the type of digital reference question 
( Measure 10 — Type of Digital Reference 
Question Received). 
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10. Type of digital reference 
questions received 

Definition: Type of reference question refers 
to the categorization of each reference 
question as one of the following: 
Bibliographic, Instructional, Literature 
Search, Other, Out of Scope, Reader’s 
Advisory, Ready Reference, Research or 
Subject Request, and Technical. These are 
defined as follows (definitions are also 
included in the Glossary): 

Bibliographic 

A bibliographic reference question is one that 
relates to any aspect of authorship or 
publication of a work. Bibliographic reference 
queries can include verification of a citation, 
names of authors, information about works in 
a series, edition information or copyright 
information, etc. 

Instructional 

An instructional question is one in which the 
user asks for assistance in using electronic 
resources that may be available to them and 
that may provide the answer to another 
reference question. Examples of instructional 
questions include requests for information on 
how to construct a search statement in an 
online periodical database, how to search the 
online catalog (OP AC), how to request books 
and other materials from the catalog, how to 
limit searches by domain in a particular 
search engine, and how to use Boolean Logic. 

Literature Search 

A literature search is a request for all of the 
published literature on a given topic or by a 
given author. The literature search may be 
limited by such factors as date, place of 
publication, peer-reviewed journals only, etc. 



Other 

For the purposes of this manual, “other” 
refers to those questions that are within the 
scope of the service but do not fit into any of 
the other categories. Digital reference 
questions that fit more than one category 
should be classified as “other.” 

Out of scope 

An out of scope question is one that will not 
be answered by the digital reference service 
because it does not meet the criteria set by the 
service for provision of an answer. Out of 
scope questions are often referred to another 
service within the organization or to an 
outside agency or service. 

Reader's Advisory 

A reader's advisory question refers to requests 
for information regarding material the user 
would like to read. Reader's advisory 
questions often take the form of asking for 
similar books by plot, other books by an 
author, other books in a series, availability of 
works in a specific format (e.g., large type, 
book on CD or tape), works in a particular 
language, or information about the 

background of a particular book. 

Ready Reference 

Ready reference (or quick fact) questions are 
those that usually have a single, finite answer. 
The answers can generally be found in 
common reference works such as almanacs, 
encyclopedias, directories, dictionaries, 

atlases, thesauri, and factbooks. Some 
examples of ready reference questions 
include: What is the population of Indonesia? 
What is the capital of Brazil? On what date 
did Queen Victoria die? How do you spell 
“symbiotic”? It should be noted that many 
digital reference services will only answer 
ready reference type questions. 
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Research or Subject Request 

A research question is one in which the user 
requests a variety of information on a 
particular topic. The research question will 
most likely have many components to the 
answer (i.e., articles from journals, books, 
citations, essays, statistics, raw data) and the 
answer may consist of responses sent in many 
formats (emailing of fiill-text articles or 
citations, pushing websites, documents or 
spreadsheets, image files, video clips, etc.). 

Technical 

A technical question is defined as a reference 
question in which the user asks for assistance 
in the use of the technology required to access 
the digital reference service or other aspects 
of accessing the library’s or organization’s 
website. These questions can include: How do 
I download Adobe Acrobat Reader? How do I 
open attachments? How do I install chat 
software? 

Rationale: Knowing what types of questions 
are asked through the digital reference service 
will assist in making collection development 
decisions regarding the acquisition of 
electronic and print (if used) reference 
resources. Having an accurate assessment of 
the type of reference questions asked will also 
impact decisions regarding assignment of 
staff in answering digital reference questions 
based on their area of expertise or 
specialization. Additionally, a clear 
understanding of the types of reference 
questions asked can assist in the design of the 
reference service components. For example, if 
many questions are ready reference type or 
repeat questions with simple answers, a 
frequently asked questions (FAQ) web page 
may be developed, or an archive of previously 
asked questions could be mounted. If a 
number of bibliographic questions are asked, 
more bibliographic resources may be needed. 
If technical questions about how to use the 
electronic resources are frequently asked, then 



a staff member who excels in giving step-by- 
step instruction or who understands Boolean 
logic may be called upon to answer such 
questions. Analysis of technical questions 
may also result in modifications, 
improvements and upgrades to the system. 

Data Collection Procedure: Continuous 
collection or Sampling. 

Collected by: Staff and manager or 
supervisor. 

Frequency: Logs are kept daily. Data is 

collected and tabulated in two-week 
periods and is generally reported annually 
unless required more frequently. 
Procedures: Staff will be instructed in 

the proper classification of digital 
reference questions by type, which will 
include the following categories (See 
above or glossary for definitions): 

• Bibliographic 

• Instructional 

• Literature Search 

• Other 

• Out of Scope 

• Reader's Advisory 

• Ready Reference 

• Research or Subject Request 

• Technical 

Continuous collection: Upon receipt 
of each digital reference question, 
staff will determine what type of 
question it is based on the definitions 
above. Staff will record the type of 
question received in the reference 
transaction log. Every two weeks the 
data will be collected or tabulated by 
type of digital reference question 
received. 

Sampling: A typical two-week period 
is selected and each digital reference 
question received is assigned a type 
based on the categories above. The 
data is tabulated and collected at the 
end of the two-week period and the 
results are extrapolated for the rest of 
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the year. It is recommended that 
sampling be done several times 
throughout the year and that 
considerations be given to time 
periods of peak and diminished use. 

NOTE: If a digital reference question can 
be classified in more than one category, 
count the question in the “other” category. 

Helpful Tools - See Appendix I for: 

H. Sample Digital Reference Transaction 
Record. 

N. Type of Digital Reference Question 
Received. 

Issues and Considerations: This measure 
requires that a detailed record be kept of the 
type of reference questions asked and that 
each question be assigned to a particular 
category. Some questions may have multiple 
type responses or may not fit one of the 
defined categories. These questions should be 
designated as “other.” Commercial reference 
tracking software may not include these 
categories for easy collection. Manual logs 
may need to be developed to incorporate this 
component. Additionally, because of the time 
consuming nature of tabulating this 
information, a sample period may be 
measured. One of the drawbacks of sampling 
is that an accurate assessment may be missed 
due to certain types of questions being asked 
at certain times of the year. This may be 
particularly true in an academic setting where 
there may be more bibliographic and 
instruction type questions at the beginning of 
a semester due to the arrival of new students 
and more research type questions toward the 
end of the semester when papers are due. To 
some extent these same fluctuations may be 
apparent in a public library setting. 
Increasingly, many public library users are K- 
.12 students or college students who are 
participating in distance learning programs 
and rely on their public libraries because they 



do not have physical access to their college 
resources. Any sampling should take into 
account these variations. 

ll.Number of referrals 

Definition: The number of referrals is a count 
of the number of users who have been 
referred to another internal contact point 
within the library or digital reference service 
(e.g., the regular reference desk or special 
section of the library such as genealogy, local 
history, government documents) or to an 
external contact point (e.g., another library or 
outside agencies such as governmental 
departments, medical or health facilities, legal 
organizations). The referral can be made for a 
number of reasons: non-affiliated users, 
inappropriate question for service (e.g., 
service only answers ready reference type 
questions) or the lack of resources necessary 
to answer the question. Do not count as a 
referral information provided to the user 
regarding possible resources, such as 
electronic databases or websites, which the 
user can access to find the answer to the 
question independently. Providing this kind of 
information to the user is counted as a Digital 
Reference Answer as defined in Measure 3 - 
Number of Digital Reference Answers. 

Rationale: The number of referrals is an 
important measure that can be used to assist 
in collection development. If a lot of referrals 
are being made, it may mean that the 
collection (either print or electronic) that is at 
the disposal of the staff is inadequate. This 
measure can also help with planning in terms 
of the scope of service offered. In the case of 
internal referrals, this measure can also be 
used to determine staffing needs for the 
digital reference service and for other 
departments to which the referrals have been 
made (e.g., traditional reference, technical 
services, bibliographic or computer training 
instruction within the library). In addition, an 
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analysis of the external referrals may provide 
opportunities for cooperation and 
collaboration with other libraries, agencies, or 
organizations. 

Data Collection Procedure: Continuous 
collection or Sampling. 

Collected by: Staff and manager or 
supervisor, electronically generated 
transcripts. 

Frequency: Logs are kept daily. Data is 

collected and tabulated in two-week 
periods or monthly periods and is 
generally reported annually unless 
required more frequently. 

Procedures: Instruct staff on the 

definitions of internal and external 
referrals and correct procedures for 
logging the referral. Inform staff of any 
special considerations they should take 
into account. 

Continuous collection: Logs are kept 
daily or transcripts are utilized if 
available. Data is collected and 
tabulated every two weeks or monthly, 
and is generally reported annually 
unless required more frequently. 
Sampling: Select a typical two-week 
period or monthly period, keep count 
on a daily basis and then extrapolate 
based on the sample data. Collect data 
for a sample two-week period or 
monthly period at various times 
throughout the year to account for 
periods of peak or diminished use. 

This measure should be used in 
conjunction with the following measures: 

3 - Number of Digital Reference Answers. 

Helpful Tools - See Appendix I for: 

H. Sample Digital Reference Transaction 
Record. 

M. Sample Digital Reference Data 

Collection Worksheet - Compilation. 



Issues and Considerations: Keeping track of 
the number of referrals will require the 
maintenance of a manual log to tally the 
number of referrals and to reflect whether the 
referral was internal or external. 

Electronically generated transcripts can also 
be used if available. In either case, careful 
analysis of the number and nature of referrals 
is crucial. It is also important to get a sense of 
why the question was referred and 
specifically where within the internal or 
external referral network the user was 
referred. This level of detail will require more 
diligence and effort on the part of the staff 
member keeping the log. Another issue is 
determining whether the referral resulted from 
the digital reference service not having the 
resources that they should be expected to have 
as compared to possible misunderstanding by 
the user about what services the digital 
reference service can provide. This analysis 
may result in very different management 
decision and actions. For example, if the user 
submits a complex reference question and is 
referred to the traditional reference service 
because the scope of the digital reference 
service has been defined as only the provision 
of ready reference, the managerial response 
may be to alter the advertising of the service 
to make this limitation clear. If a question is 
submitted that is within the scope of the 
service, but a referral is made because the 
service has no resources to answer the 
question, the issue becomes one of collection 
development and perhaps reallocation of 
resources. Finally, referrals may be made 
when the question could have been answered 
by the digital reference service. This could 
indicate a need for more training or that there 
is not enough staff to handle the workload. 
The level of detail kept in the log may help to 
determine if this is the case. 
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12. Saturation rate 

Definition: The saturation rate is the measure 
of the ratio of digital reference service users 
who are members of the target population to 
the total number of members of the target 
population. 

Total digital reference service users* X 100 
Total number of target population 

*who are members of target population 



To calculate this measure it is first necessary 
to determine the number of digital users and 
the population of the target group. 

Rationale: Saturation rate is an important 
measure that reflects how many members of 
the given target population actually use the 
digital reference service. This percentage can 
assist in decision-making processes regarding 
advertising of service, importance of service 
to the target population, and subsequent 
allocation of resources to the digital reference 
service. 

NOTE: Some libraries may not be able to 
complete this measure as they may be unable 
to determine the total number of members in 
the target population group. 

Data Collection Procedure: Other: 
demographic analysis. 

Collected by: Manager or Supervisor. 
Frequency: Monthly or annually. 

Procedures: Determine the total number 

of users of the digital reference through an 
analysis of the logs. Determine the target 
service population. Divide the total 
number of digital reference users who are 
part of the target service population, by 
the total target population and multiply by 
100 . 



Helpful Tools - See Appendix I for: 

M. Sample Digital Reference Data 

Collection Worksheet - Compilation. 

Issues and Considerations: Determining the 
target population may be relatively simple as 
in the case of an academic setting in which 
the target population is the faculty, staff, and 
students of the college, school or university, 
or in a public library setting where the 
“chartered to serve” area and population are 
clearly defined and accurately match the 
actual service area in practice. It may be more 
difficult to determine the target service 
population in cases where the digital 
reference service is strictly virtual and the 
target population is so vast or ephemeral as to 
be not useful, or in a public library setting in 
which the chartered to serve population area 
is not clearly defined and does not match the 
actual service area in practice. Also, it may be 
difficult to determine if an actual user is, in 
fact, part of the target population. Again, 
some libraries may not be able to complete 
this measure as they may be unable to 
determine the total number of members in the 
target population group. 

13. Sources used per question 

Definition: This statistic will identify each of 
the resources used in responding to a digital 
reference question and will include but not be 
limited to such resources as: full-text and 
citation/abstract databases; statistical 
databases; directories; bibliographic 
resources; electronic reference books and 
encyclopedias; e-joumals and e-text 
subscriptions; image archives; local and 
regional OPACs and consortial catalogs (such 
as OCLC’s WorldCat) and aggregate 
databases that can include all of the above 
(such as Dialog and Lexis-Nexis, OCLC’s 
FirstSearch); webpages and other internet 
resources; newsgroup resources; listserv 
resources; and scanned resources transmitted 
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electronically. Also include here traditional 
(print-based) resources that are used to 
answer questions, so long as the answer 
obtained from the print resource is transmitted 
to the user electronically. Finally, include 
here resources that have been created by the 
staff. Depending on the level of detail desired, 
each of the resources listed above can be 
designated individually and will also be 
designated as belonging to one of the 
following categories: Free (electronic), Staff- 
Created (electronic), Proprietary (vendor- 
based electronic), Traditional (print-based) 
and Other. 

Rationale: “Sources used” is an important 
descriptive measure that will have a decided 
impact on broad-based decisions concerning 
allocation of resources both for the digital 
reference service and for the library as a 
whole. Additionally, determination of sources 
used can also serve to assist in collection 
development decisions regarding the 
percentage of budget allocated to proprietary 
or fee-based resources and print resources 
based on the availability of comparable free 
resources. 

Data Collection Procedure: Manual Log 
Analysis. 

Collected by: Staff; tabulated by 
designated staff or manager or supervisor, 
Frequency: Log kept daily, tabulated 

and collected monthly, reviewed and 
reported annually. 

Procedures: You may choose from one 

of two possible approaches to this 
measure depending on the level of detail 
desired: 

1 . If a high level of detail is required or 
desired, the resources used can be 
described in detail using a manual log 
sheet and will include those 
designations as described in the 
definition above in addition to the 
additional designation of Free, Staff- 



Created, Proprietary, Traditional and 
Other. 

2. If a high degree of detail is not 
required the sources used may be 
designated as simply Free, Staff- 
Created, Proprietary, Traditional or 
Other on a check list. 

NOTE: Only count staff-created resources 
in that category. Do not also count in the 
free category. 

Helpful Tools - See Appendix I for: 

H. Sample Digital Reference Transaction 
Record. 

O. Sources Used Per Question. 

Issues and Considerations: Depending on 
your situation and needs, you may choose to 
keep more detailed and specific data to track 
the types of resources used, or just complete 
the tally sheet using the five broad categories 
on a checklist. Knowing why you want to use 
the statistic and the time and staff you have 
available to collect this information will have 
an impact on your decision. If you want to do 
a high level analysis of the use of resources 
by subject area in addition to breakdown by 
type of resource, the detailed approach is 
appropriate and useful. 

14. Repeat Users (Return Rate) 

Definition: This statistic represents the 
number of users who utilize the digital 
reference service more than once. 

Rationale: The number of repeat users (or 
return rate) is a measure that can indicate 
satisfaction with the service. The return rate 
can also be used in conjunction with other 
measures to determine a relative ratio of new 
users to return users. 

Data Collection Procedure: 

Other: Manual Log Analysis. 
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Web Log Analysis. 

Digital Reference Tracking Software Report. 
Collected by: Designated staff member 
or manager or supervisor. 

Frequency: Monthly. 

Procedures: Make arrangements to 

receive logs and reports from the digital 
reference staff and technology department 
if necessary. Receive training in reading 
and analysis of the electronic logs and 
reports. Analyze reports to determine the 
number of users of the service who are 
repeat users. 

This measure should be used in 
conjunction with the following measures 
to obtain a more comprehensive and 
meaningful assessment of how “repeat 
use” informs the assessment process. 

24 — Reason for Use. 

Helpful Tools - See Appendix I for: 

H. Sample Digital Reference Transaction 
Record. 

M. Sample Digital Reference Data 

Collection Worksheet - Compilation. 

Issues and Considerations: Determining the 
repeat user is one of the most important 
measures to collect because it can be an 
indication of user satisfaction with the 
service. It can, however, be tricky to collect 
and even trickier to analyze. Analysis of 
repeat users should be done in a sensitive and 
careful manner. It should not be assumed that 
if users do not use the service more than once 
that they were dissatisfied with their first use. 
Conversely it should not be assumed that 
repeat users are satisfied with the service (in 
fact, the digital reference service may be the 
only service available to the user). Some 
issues that might arise when collecting the 
data for this measure include the following: 

• The time frame that may elapse between 
first and subsequent uses of the digital 
reference service. For example, it may take 



weeks, months or years for a person to use the 
service more than once. 

• Tracking and keeping record of repeat 
users may be difficult. If the digital reference 
service is small enough, users’ names may be 
kept in a manual log. 

• Commercial reference question tracking 

and management software (e.g., 24 / 7 , 

QuestionPoint, LSSI’s Virtual Reference) or a 
reference question database developed by the 
digital reference service will provide a better 
electronic measure of this statistic than using 
weblogs generated from web usage tracking 
software (e.g., Webtrends). In many cases (as 
in the case of several people accessing the 
service from the same machine which 
happens in a physical library setting, or users 
submitting questions anonymously), an 
electronically generated log alone will not 
give an accurate measure of repeat users. In 
many cases anonymous access to the digital 
reference service is permitted, which would 
prevent collection of completely accurate 
information. This measure may need to be 
combined with a more detailed manually kept 
log of users. 

• Finally, this measure will not give an 
accurate qualitative analysis of why users use 
the service more than once. This measure can 
be coordinated with the information gathered 
in Measure 24 - Reason for Use, which 
reflects the actual reasons for return use for a 
more comprehensive and meaningful 
assessment. 
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LOG AND REPORT 
ANALYSIS 

IMPORTANT NOTE: For the purposes of this 
section, which includes Measures 15 through 
19, log analysis refers to an analysis of data 
electronically generated by a software 
program designed to track traffic on a website 
or email server. Examples of such software 
include Web trends http://www.webtrends.com 
and Webtracker http://www.fxweb.com/ 
tracker. Report analysis refers to 
electronically generated reports, data and/or 
information that can be provided by 
commercial digital reference software such as 
LSSI’s VRS http://www.lssi.com, 24/7 
http://www.247ref.org, QuestionPoint or 
customized reference databases (created in 
database software such as MS Access). Please 
note that each of these commercially 
available resources has specific and 
predefined definitions for the terminology 
used. Of particular interest for the purposes 
of this manual are definitions related to such 
terms as “sessions, ” “browser, ” and 
“platform”. The word “ sessions ” for the 
purposes of this manual refers only to the 
number of sessions registered in an 
electronically generated report. A “ browser ” 
refers to the software program that is used to 
navigate the Internet (Netscape Navigator, 
Internet Explorer, etc.). " Platform ” refers to 
an operating system, or underlying the 
software program that permits a computer 
system to function (such as Windows NT, 
Windows 2000, Mac OS, Linux). Also note 
that electronic reports (as in the case of Link 
Analysis) can be generated by utilizing well 
known Internet search tools such as Google. 
For the purposes of this section log analysis 
does not refer to manually kept logs or tally 
sheets. 



15. Number of digital reference 
sessions 

Definition: The total number of sessions 
logged using a digital reference service web 
page, real time chat software, or interactive 
digital video as determined by electronic log 
analysis. Do include sessions at the digital 
reference service FAQ and archives if these 
pages exist. Do not include sessions to the 
library’s home page or other pages within the 
library website that do not relate directly to 
the digital reference service or provide a 
direct link to the digital reference service. 
Also, do not include sessions as determined 
by a manual log. 

Rationale: An analysis of digital reference 
sessions as documented by the log provides 
an approximation of the number of visits to 
the virtual library and can be seen as 
analogous to the traditional measure “number 
of people who use the traditional library 
reference/information desk.” This measure 
can be useful in determining staffing 
requirements. As one field tester stated, “this 
is a crucial statistic for gauging level of use 
trends that are often used for budget/staffing 
decisions. As long as one realizes that the 
measure is not a perfect one and how it is 
imperfect, the [electronically generated] 
system supplied number can be an efficient 
relative measure of use.” Thus increases and 
decreases in the number of sessions over time 
can be used to assess changes in the digital 
reference service that may occur for a number 
of reasons. By adding this count to users of 
the traditional reference/information desk, a 
more accurate analysis of total reference use 
may be estimated. 

Data Collection Procedure: Log Analysis. 
Collected by: Electronically generated 
report and/or log provided by vendor, but 
viewed and analyzed by staff and/or 
manager or supervisor. May need to 
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request information from the technology 
department. 

Frequency: Monthly. Reported 

annually. 

Procedures: Make arrangements with 

vendor and technology department if 
necessary to receive reports and logs on a 
regular basis. Receive appropriate training 
in analyzing the logs. Determine the 
number of digital reference sessions from 
the logs or report. 

Each vendor and type of software used to 
generate reports of activity may mean 
something different when defining 
sessions. Also, different software products 
may use a number of data points that 
reflect activity. For example some chat 
software products will give a report for 
“sessions requested” and “sessions 
honored.” 

Helpful Tools - See Appendix I for: 

J. Digital Reference Question 
Transaction Log - Real Time 
Sessions. 

Issues and Considerations: “Sessions” as a 
measurement tool reported by electronically 
created logs refers primarily to real-time 
interactive services such as chat, voice over 
IP, and interactive video. The number of 
digital reference sessions as an analogous 
measure to physical library reference desk 
visits is not a strict analogy. Many people 
may log in during the same session. The 
number of sessions may not be proportionate 
to the number of reference questions received 
as multiple questions may be submitted 
during one session, multiple users may be 
logged in during the same session (as is the 
case when the question is submitted from a 
computer within the library or organization). 
Additionally, and as importantly, different 
software products define, measure, and report 
sessions differently. It was recommended by 



several of the field testers and members of the 
advisory committee that libraries and digital 
reference services engage in dialogue with the 
vendors of such software products to develop 
a standard for defining the number of 
sessions. 

16. Usage of digital reference 
service by day of the week 

Definition: Usage of service by the day of the 
week tracks the use of the digital reference 
service on each of the days of the week over a 
period of time. 

Rationale: This measure will be useful in 
determining usage, (high volume, average, 
low volume) by day of the week and as a 
result will be useful in the scheduling of staff 
for the digital reference service. 

Data Collection Procedure: Log Analysis. 
Collected by: Provided by vendor of log 
or digital reference software. Collected by 
designated staff member or manager or 
supervisor. 

Frequency: Monthly. 

Procedures: Make arrangements to 

receive logs and reports from the 
technology department if necessary. 
Receive training in reading and analysis 
of logs and reports. Analyze logs to 
determine usage by tabulating the number 
of reference questions received by day of 
the week. 

If using weblog software, it may be most 
useful to look at statistics generated 
regarding the page on which the reference 
question submission form or email link 
appears. 

This measure can be used in conjunction 
with: 

17 - Usage of digital reference service by 
time of day. 
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Helpful Tools — See Appendix I for: 

C. Sample Webtrends Report — Activity 
by Day of the Week 

H. Sample Digital Reference Transaction 
Record. 

Q. Log Analysis - Usage by Day of the 
Week / Usage by Time of the Day. 

Issues and Considerations: The analysis of 
the weblogs and reports generated 
electronically will not be able to determine if 
more than one question was submitted in a 
single transmission. Another consideration is 
that the analysis may show that a high number 
of sessions occur when the physical library is 
closed, and decisions will need to be made 
regarding the provision of service on such 
days. Such analysis may also show that 
turnaround times (predetermined target times 
for the provision of an answer after the 
question has been submitted) may need to be 
evaluated in light of the data received from 
the logs. This measure should not be the only 
criteria used for staffing considerations. The 
next measure ( Measure 17 - Usage of digital 
reference service by time of day) should also 
be taken into account, as should an analysis of 
other duties performed by the digital 
reference staff both within the digital 
reference service (i.e., creation of pathfinders 
and FAQ's) and in other areas of the library 
(since in many cases digital reference staff 
also perform other library duties). It should 
also be noted that, because of the limitations 
of weblog software and to a lesser extent 
reports generated by digital reference 
software, this measure and the following 
measure will not give the entire picture of use 
of digital reference service, and should be 
used in combination with other data collection 
methods (i.e., manual logs and records). 



17. Usage of digital reference 
service by time of day 

Definition: Usage of service by time of day 
tracks the number of reference questions 
received in hour increments over the course of 
the day, and is tracked over a period of time. 
This statistic will be useful in determining 
peak usage times and slower times. 

Rationale: This measure will be useful in 
determining usage (high volume, average, low 
volume) by the time of day, and as a result 
will be useful in the scheduling of staff for the 
digital reference service. 

Data Collection Procedure: Log Analysis. 
Collected by: Provided by vendor of log 
or digital reference software. Collected by 
designated staff member or manager or 
supervisor. 

Frequency: Monthly. 

Procedures: Make arrangements to 

receive logs and reports from the 
technology department if necessary. 
Receive training in reading and analysis 
of logs and reports. Analyze logs to 
determine usage by tabulating the number 
of reference questions received by hour of 
the day. 

If using weblog software, it may be most 
useful to look at statistics generated 
regarding the page on which the reference 
question submission form or email link 
appears. 

Other useful measures related to this 
measure: 

16 - Usage of digital reference service by 
day of the week 

Helpful Tools - See Appendix I for: 

D. Sample Webtrends Report - Activity 
by Hour of Day. 
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H. Sample Digital Reference Transaction 
Record. 

Q. Log Analysis - Usage by Day of the 
Week / Usage by Time of the Day. 

Issues and Considerations: This measure 
should not be the only criteria used for 
staffing considerations. The preceding 
measure ( Measure 16 - Usage of digital 
reference service by day of the week) should 
also be taken into account, as should an 
analysis of other duties performed by the 
digital reference staff both within the digital 
reference service (i.e., creation of pathfinders 
and FAQ's) and in other areas of the library 
(since in many cases digital reference staff 
also perform other library duties). Another 
consideration is that the analysis may show 
that a high number of sessions occur during 
time periods when the physical library is 
closed and decisions will need to be made 
regarding the provision of service during such 
times. It should also be noted that, because of 
the limitations of weblog software and to a 
lesser extent reports generated by digital 
reference software, this measure and the 
preceding measure will not give the entire 
picture of use of digital reference service, and 
should be used in combination with other data 
collection methods (i.e., manual logs and 
records). 

18. User’s browser 

Definition: The user’s browser is defined as 
the type of web browser (internet navigation 
software program) employed by the user to 
access the digital reference service (e.g., MS 
Internet Explorer, Netscape, AOL Browser). 

Rationale: Knowing the user’s browser can 
help in the analysis of technical difficulties 
that may arise when the user submits the 
reference query. This knowledge may also be 
useful to the staff when they push websites to 
the patron, explain login procedures for 



databases to the user or encounter other 
situations in which they need to “walk” the 
patron through a navigational process. 
Additionally, an overall analysis of the users’ 
browsers can be used in the development and 
design of the digital reference website. Ideally 
the digital reference website interface should 
be compatible with a number of operating 
systems and platforms. 

Data Collection Procedure: Log Analysis. 
Collected by: Provided by vendor of log 
or digital reference software. Collected by 
designated staff member or manager or 
supervisor. 

Frequency: Monthly. 

Procedures: Make arrangements to 

receive logs and reports from the 
technology department if necessary. 
Receive training in reading and analysis 
of logs and reports. Analyze logs to 
determine usage by type of browser. 

Other useful measures related to this 
measure: 

19 - User’s platform. 

Helpful Tools - See Appendix I for: 

H. Sample Digital Reference Transaction 
Record. 

R. Log Analysis - User's Browser / 
User's Platform. 

Issues and Considerations: The level of 
detail provided by the log analysis may 
include detailed information on not only the 
type of browser that the user is employing to 
access the service, but also the version. There 
can be great variations in capabilities and 
features among different versions of the same 
browser, and analysis of this data, which can 
be translated into information that can be used 
to assist in technical problems and questions, 
will require staff to have a sophisticated level 
of knowledge about the variations in 
browsers. This level of expertise may require 
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additional training of existing staff, the hiring 
of additional staff, or a collaborative or 
cooperative agreement between the digital 
reference service and technology team of the 
library or organization. 

19. User’s platform 

Definition: User's platform is defined as the 
type of operating system employed by the 
user to access the digital reference service 
(e.g., Windows 98, Windows 2000, Windows 
NT, Windows Millennium, Windows XP, 
Linux, Macintosh OS). A further level of 
detail is also possible that will reflect which 
release and version of the platform is being 
used. 

Rationale: Knowing the user’s platform or 
operating system can help in the analysis of 
technical difficulties that may arise when the 
user submits the reference query. This 
knowledge may also be useful to the staff 
when they push websites to the patron, 
explain login procedures for databases to the 
user or encounter other situations in which 
they need to “walk” the patron through a 
navigational process. Additionally an overall 
analysis of the users’ browsers can be used in 
the development and design of the digital 
reference website. Ideally the digital reference 
website interface should be compatible with a 
number of operating systems and platforms. 

Data Collection Procedure: Log Analysis. 
Collected by: Provided by vendor of log 
or digital reference software. Collected by 
designated staff member or manager or 
supervisor. 

Frequency: Monthly. 

Procedures: Make arrangements to 

receive logs and reports from the 
technology department if necessary. 
Receive training in reading and analysis 
of logs and reports. Analyze logs to 
determine usage by user’s platform. 



Other useful measures related to this 

measure: 

18— User’s browser. 

Helpful Tools - See Appendix I for: 

H. Sample Digital Reference Transaction 
Record. 

R. Log Analysis - User's Browser / 
User's Platform. 

Issues and Considerations: The level of 
detail provided by the log analysis may 
include detailed information on the version of 
the platform that the user is employing to 
access the service. There can be great 
variations in capabilities and features among 
different versions of the same type of 
platform (i.e., significant differences between 
the various Windows platforms), and analysis 
of this data, which can be translated into 
information that can be used to assist in 
technical problems and questions, will require 
staff to have a sophisticated level of 
knowledge about the variations in platforms. 
This level of expertise may require additional 
training of existing staff, the hiring of 
additional staff or collaborative or cooperative 
agreement between the digital reference 
service and technology team of the library or 
organization. 
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USER SATISFACTION 
MEASURES 

20. Awareness of service 

Definition: Awareness of service is a measure 
of how aware the target population is of the 
availability of the service. 

Rationale: Determining the target 

population’s awareness of the service is 
useful in developing marketing strategies for 
the service. It can also be used in conjunction 
with other user satisfaction measures to 
determine if light or heavy use of the service 
is a result of awareness of the service to begin 
with or a result of satisfaction with the service 
once it has been used. 

Data Collection Procedure: 

Survey. 

Questionnaire. 

Interview. 

Focus Groups. 

Collected by: Staff and management or 
supervisor. 

Frequency: Annually. 

Procedures: Determine target popu- 

lation parameters. Determine how best to 
survey or interview representative sample 
of target population. Develop 
survey/interview instrument, which may 
include questions related to other 
measures. Decide method for 
administering survey or interview (phone, 
paper, electronically, in person). 
Administer survey. Analyze results. 
Review survey/interview instrument for 
possible improvements. 

Helpful Tools - See Appendix I for: 

T. User Satisfaction Measures — Sample 
Survey/Questionnaire Questions 

Issues and Considerations: Determining the 
general awareness of the digital reference 



service by the target population requires that 
surveys or interviews be conducted across the 
target population base. The survey must 
include those who have used the service and 
those who have not. Selecting a sample group 
of participants that will provide statistically 
significant and meaningful results may 
require a higher level of understanding of 
statistics and research methods than may be 
possessed by the library or organization. The 
library may consider hiring an outside 
consultant to assist with this measure. The 
gathering of information for this measure will . 
require the development of a survey, 
questionnaire, or interview instrument that 
will get at the information that is needed or 
desired. 

This measure will also require that contacts be 
made outside the library or organization and 
may require collaborative agreement with 
other individuals, departments, or agencies 
who can assist in the distribution and 
collection of the surveys. For example, in a 
public library setting, combining this survey 
with those that may done by other city or 
county agencies to assess citizen awareness 
and satisfaction levels may be appropriate 
(and cost effective). In an academic setting 
collaboration could be with other academic 
services such as admissions and records, 
student services, and counseling. We are 
aware that the level of effort required for 
gathering these measures may be high, 
however, we feel that this is an important 
measure for determining the quality of service 
and suggest that any attempts at getting a 
sense of the general awareness of the service 
will provide useful information and will be 
much better than having no information at all. 

Focus groups may be useful for this measure 
if the geographic area in which the target 
population resides is relatively small (for 
example, an academic library or public library 
setting), but may be less useful for digital 
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reference services that serve a less-well 
defined area or the population at large (e.g., 
AskERIC, QuestionPoint, or the Internet 
Public Library Digital Reference Service). 

21. Accessibility of service 

Definition: Accessibility of service is a 
measure of how easily potential users are able 
to avail themselves of the service and includes 
(but is certainly not limited to) such factors 
as: availability (both time and day of the 
week); site design (simplicity of interface); 
ADA compliance; ease of use; placement in 
website hierarchy if using web submission 
form or email link from the website; use of 
metatags for digital reference websites 
(indexed in major search tools, etc.); or 
multilingual capabilities in both interface and 
staff, if warranted based on target population. 

Rationale: An understanding of the 

accessibility of the service can assist in 
management decisions regarding scheduling 
of staff, design of the digital reference service 
website, design of the library website, the 
need for adaptive technology for users with 
special needs, and the need for multilingual 
staff, if warranted. 

Data Collection Procedure: 

Survey. 

Questionnaire; 

Interview. 

Other: testing of accessibility by staff and/or 
selected group of users and potential users. 
Review of accessibility issues by expert 
consultants. 

Collected by: Staff and Management or 
Supervisor. 

Frequency: Annually. 

Procedures: Determine target popu- 

lation parameters. Determine how best to 
survey or interview representative sample 
of target population. Develop survey/ 
interview instrument, which may include 



questions related to other measures. 
Decide method for administering survey 
or interview (phone, paper, electronically, 
in person). Administer survey. Analyze 
results. 

Each of the potential factors, as identified 
in the definition above, should be viewed 
as separate points of assessment, using a 
variety of the recommended procedures. It 
is not recommended that these factors be 
assessed together using a single 
instrument or procedure. 

This measure will require contacting both 
users and potential users of the service. 
The complexities of sampling for this 
measure can be simplified by identifying 
specific target populations within the 
general target population. For example, 
potential and actual users who may have 
special needs due to visual or hearing 
differences could be approached for 
participation in the assessment process by 
contacting local support groups. Distance 
learning students could be contacted 
through their advisors. Cultural groups in 
the community or on campus can be 
contacted regarding language consider- 
ations. 

As of this writing, if the point of service 
for accessing the digital reference service 
is a website (either designed and created 
by a vendor or custom designed by the 
library or digital reference service), a 
measure of the degree of accessibility 
with regard to Americans with Disabilities 
Act requirements can be obtained by 
utilizing “Bobby.” This website provides 
a utility in which any web address (URL) 
can be entered, and a report will be 
generated that explains any problems the 
website entered has regarding 
accessibility for people with visual and 
auditory difficulties. The current web 
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address for the Bobby site is 
http://www.cast.org/Bobby. 

Finally, it may, in fact, be more cost 
effective to hire expert consultants on 
such topics as website placement and 
ADA regulations to review the digital 
reference service accessibility components 
and make recommendations for 
improvements. 

Issues and Considerations: Survey, 

interview, and testing instruments that will 
address different accessibility issues must be 
developed. The administration of these 
assessment tools can be time consuming and 
may require a level of expertise and 
knowledge of statistical and research methods 
that the organization may not have, so hiring a 
consultant may be the most effective, efficient 
and cost-effective method of deriving the 
information for this measure. In the case of 
digital reference services affiliated with 
academic libraries it may be possible to 
consult with the many research offices or 
institutes that exist to assist with the 
assessment. Public libraries may be able to 
collaborate with a local university’s research 
offices or institutes for assistance. The authors 
realize however, that this may be beyond the 
capabilities or budgets of some digital 
reference services and encourage the service 
to make an attempt at gathering data for this 
measure. It is also important if accessibility 
for special populations (i.e., Hispanics or 
speakers of other languages or those with 
special visual, auditory or mobility needs) is 
an issue that members of these populations be 
consulted or included in the survey, interview, 
or testing processes. 

22. Expectations for service 

Definition: Expectations for service measures 
levels of service users think they will be 
getting when they submit a question. A 
variety of expectations can be measured 



including the following (see the Glossary for 
definitions): 

• Types of questions that will be answered. 

• Format of answers. 

• Length of time for the question to be 
answered. 

(see also Measure 8 — Digital Reference 
Completion Rate) 

• Degree of commitment to finding an 
answer. 

• Correct, definitive or comprehensive 
answer. 

Rationale: In many cases, perception is 
everything. What the users expect with regard 
to the service can have a decided impact on 
their assessment of the quality of service. 
Understanding a user’s expectations in each 
of these areas will assist in the ongoing 
development of the digital reference. These 
measures can also be used to analyze the 
marketing of the digital reference service to 
determine if the services advertised clearly 
inform the user of the scope of the service. 
See the Issues and Considerations section for 
a more detailed description of each of these 
expectations. 

Data Collection Procedure: 

Survey. 

Questionnaire. 

Interview. 

Focus Groups. 

Collected by: Staff and Management or 
Supervisor. 

Frequency: Varies, can be conducted 

annually or on ongoing basis. 

Procedures: Determine how best to 

survey or interview representative sample 
of users. 

Develop survey, interview or 
questionnaire instrument, which may 
include questions related to other 
measures. 

Decide method for administering survey, 
questionnaire or interview. For example, it 
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may be sent by mail or email, accessed 
electronically (via website or template 
survey sent out with each final response), 
or conducted by phone or in person. 
Determine return date for survey if 
necessary. 

Administer survey. 

Tabulate and analyze results. 

Conducting surveys and interviews of 
users after they have used the service can 
be problematic in that it may be difficult 
to track them down or to get them to agree 
to participate. Additionally, if the 
assessment is done after the service is 
provided, initial expectations held by the 
user prior to using the service may be 
colored by their actual experience with the 
service. It would be ideal to incorporate a 
short questionnaire or survey periodically 
either at the beginning of the reference 
transaction (as part of the reference 
interview that can be conducted via email, 
digital reference submission form or at the 
beginning of a real-time session), or close 
to an end of the reference transaction. It is 
also useful to make the completion of the 
survey or the interview process as simple 
as possible for the user. 

Focus groups are an ideal way to collect 
the information needed for this measure if 
the potential participants are able to meet 
in a face-to-face setting. This may be 
possible if the potential users are 
geographically close to one another. 
However, many digital reference services 
exist in order to provide service to those 
who are not in close proximity to the 
physical library or exist to serve a global 
population. It may be possible to conduct 
virtual focus groups as a way of collecting 
information for this measure from remote 
users. 



This measure can be used in conjunction 
with: 

7 - Correct Answer Fill Rate. 

Helpful Tools - See Appendix I for: 

S. User Satisfaction Measures - 
Expectations for Service Sample 
Questionnaire 

Issues and Considerations: Even though a 
digital reference service may be limited in 
scope to the types of questions answered (for 
example the service may only answer ready 
reference questions or may decide not to 
conduct full fledged literature searches for the 
user), or by other constraints (such as posted 
turnaround time, the amount of time to be 
spent on each question or the formats in 
which answers will be provided), there will 
inevitably be potential users of the service 
who do not read the policies, advertisements, 
or entry level instructions and will submit an 
out of scope question. There will also be users 
who may, in fact, read the policies and 
limitations but may not know what they 
mean. 

Types of Questions to be Answered 

In the reference world, we often use 
terminology that may not be familiar to the 
average user. For example, will the average 
user know what “ready reference” or 
“instructional” or “literature search” means? 
Also, when a user is referred to another 
internal department within the library or to an 
external organization or agency because the 
user has asked a question that is not within the 
scope of service, it may be helpful if the 
librarian or staff member is willing to elicit 
from the user what their expectations were 
and why. This can help in the development of 
accurate instructions and advertising of the 
service. 

Format of Answers 

Knowledge of the expectations of the user 
with regard to format can help the ongoing 
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development of the service by facilitating the 
ability to provide materials in a variety of 
formats. The users’ expectations can also help 
the service to address what may be unrealistic 
expectations of the users in the development 
of their digital reference service policies. For 
example, many new users may believe that 
everything is available in full-text format or 
that everything is available on the Internet 
(including books that may only be found in 
book format on the shelves of the library), or 
that, because they are requesting information 
from the library it will always be available in 
any format for free. 

Length of Ti m e for Questions to be 
Answered 

It is critical to know what users expect in 
terms of a turnaround time in order to 
determine appropriate staffing schedules and 
also to set realistic policies (both in terms of 
expectation of the user and abilities of the 
staff to meet those expectations). Many times 
there is a discrepancy between the user’s 
expectations about how long it will take to 
answer a question and how long it actually 
takes to answer that question. In an 
information society with a media that 
increasingly promises instant access to 
information, users may expect to get answers 
to their questions by simply pressing a button. 
They may not be aware of the amount of time 
it may take to conduct searches in multiple 
databases, or they may not realize that the 
information they are seeking is not available 
in the configuration or format that they desire. 
Also, they may not care that the digital 
reference staff has received ten questions 
ahead of theirs that still need to be answered. 
In an academic or school setting especially, 
but also in the public library, the user may be 
facing deadlines (papers or bibliographies 
due, job interviews the next day) that will 
impact their perception of the timeliness of 
the response. 



It is impossible for any digital reference 
service to provide instant answers to all users 
all the time. The nature and complexity of the 
question, current workload or backlog, 
technical problems, or lack of expertise may 
all impact the amount of time it takes to 
answer a digital reference question. In 
addition, the method of communication may 
also slow things down (sending emails back 
and forth). 

Commitment to Finding an Answer 

In an ideal world it would be reasonable to 
expect that the providers of any reference 
service (digital or traditional) would exhaust 
all possibilities and “go to the ends of the 
earth” in an effort to provide an answer for 
the user. Understanding the user’s 
expectations regarding the degree of 
commitment to finding an answer will help in 
the development of policies and guidelines for 
answering reference questions. Such 
questions as the examples below can help in 
setting guidelines for the extent to which the 
service will go in the provision of an answer: 

• “Will we make phone calls or send email 
to outside organizations or individuals on 
behalf of the user in an effort to get the 
answer?” 

• “Will we set a certain reasonable length of 
time to work on each question?” 

• “Will we work on the question for X 
length of time before referring to another 
internal or external service provider?” 

• “At what point do we give up if an answer 
has not been found and how do we notify 
the patron?” 

But in the real world of reference service, 
many other factors (staffing, budget, 
resources, workload, etc.) influence how far 
the service can or will go in providing an 
answer. Determining the user’s expectations 
regarding the lengths to which the service 
provider will go in providing an answer can 
help in defining the parameters of the service. 
In some cases, the user may, in fact, expect 
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that the level of commitment will be high and 
these expectations may be unreasonable. On 
the other hand, the user may not expect the 
service to provide the level of commitment to 
locating the answer that it does in fact 
provide. Getting a sense of the range of these 
user expectations can be used to develop 
policies for commitment to finding the 
answer. 

Correct, Definitive or Comprehensive 
Answer 

Of course, both staff and the user expect that 
the answer given will be correct, definitive or 
substantially comprehensive. In many cases, 
however, definitive or comprehensive 
answers may not be easy. This may occur 
when the question is an open complex 
question with many parts or when the 
question requires a proving a negative. In 
some cases the question may be one that the 
digital reference staff is not qualified or 
equipped to answer. Some examples of such 
difficult questions include: 

• “What is the best treatment available for 
breast cancer? I want you to tell me what 
to do.” 

• “What are my rights as a non-custodial 
father?” 

• “I need a bibliography of every critical 
essay, book or article on Shakespeare’s 
sonnets ever published.” 

Of course the digital reference staff will 
provide some information to the patron, but 
they cannot tell a patron what the best 
treatment is for a disease, explain to a patron 
his/her legal rights, or assure that any 
bibliography they have provided is 
exhaustive. In some cases the answer given 
may not be the one that the patron wants to 
hear (for example, the only answer to the 
question “How can I get the state of New 
York to release information about my birth 
mother if she might not want me to have that 
information?” is that you cannot get the state 
of New York to release such information 



without the birth mother’s permission). In 
addition, some reference questions cam only 
be answered by giving various viewpoints 
about a particular issue (for example, “Which 
political entity has a more legitimate 
argument for their claims and issues, the 
Israelis or Palestinians?”). 

The providers of a digital reference service 
cannot keep such questions from being asked 
and may in fact provide resources or referrals 
to help patrons in deciding the answer for 
themselves. Examples include: suggesting 
that they discuss breast cancer treatments or 
legal issues with a doctor or attorney, or 
providing various viewpoints on opinion type 
questions. Providing circumstantial evidence 
can help users draw their own conclusions 
regarding questions that require proof in the 
negative. Regardless of the amount of useful 
information that the staff provides in response 
to such questions, it may still be perceived by 
the patron as not correct, definitive, or 
comprehensive enough. 

On the other hand, the digital reference staff 
member may not have provided a correct or 
definitive or comprehensive answer to a 
question for which such answers exist, but the 
user may believe or perceive that he or she 
has in fact received a correct, definitive or 
comprehensive answer. This situation may 
result from poor staff attitude and behavior, a 
lack of skill on the part of the staff member, a 
lack of resources, or a lack of time to give a 
full response. The staff member may not 
double-check sources or evaluate the source 
of the answer properly (especially in the case 
of answers gleaned from the internet). The 
user however, may believe that because the 
service is providing the answer, that this, in 
some way, ensures correctness, definitiveness, 
or comprehensiveness. 
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23. Other sources user has tried 

Definition: “Other sources the user has tried” 
measures what other resources the user tried 
prior to submitting a question to the digital 
reference service. These resources can 
include, but are not limited to: other digital 
reference services, traditional reference 
services, self-service via web (using search 
tools such as subject directories, search 
engines, metasearch engines, or online AskA 
services, Usenet groups, OP AC or online 
database searching, personal collections, 
subject specialists, etc.). 

Rationale: Knowing what other sources the 
user tried before submitting a question to the 
digital reference service can be critical in 
learning what information the user already 
received in order to avoid duplication of 
effort. These measures can assist the staff in 
understanding what else the user would like to 
know, can assist the staff in assessing whether 
the user’s expectations for an answer are 
realistic, and also can help staff in assessing 
the possible initial frustration level of the user 
when he or she decides to utilize the reference 
service. 

Data Collection Procedure: 

Interview: Reference Interview (email, chat, 
interactive voice or video). 

Reference Question Submission Form. 
Collected by: Staff. 

Frequency: Continuous - at time of 

reference interview or submission of 
question. 

Procedures: Since most effective 

reference transactions will include a 
reference interview that will include this 
question, the virtual reference interview is 
the most efficient and logical place to 
gather this data. Digital reference question 
submission forms should have a 
component that asks this question. 

Tabulate and analyze results. 



The procedures for collecting this measure 
can include either asking a simple open- 
ended question such as “What other 
sources have you tried?” or providing a 
checklist of possible types of resources. 
Both procedures were recommended by 
the field testers. 

Helpful Tools - See Appendix I for: 

H. Sample Digital Reference Transaction 
Record. 

T. User Satisfaction Measures - Sample 
Survey/Questionnaire Questions. 

Issues and Considerations: Knowledge of the 
sources that the user has already tried can 
help to avoid duplication of effort and can 
help the staff to determine exactly what the 
user is looking for. It is important when 
questioning the user on this issue to determine 
if he or she felt the other information they 
found was useful. On the other hand, it may 
be helpful to not assume that the patron knew 
how to effectively search the resources 
already tried. It might be useful to determine 
the user’s level of sophistication in searching 
online information on his or her own (though 
this must be done delicately). If it is apparent 
that the user does not know how to effectively 
search a resource, the search might be 
conducted again by the staff. By 

understanding what resources the user has 
already tried, either successfully or 

unsuccessfully, the digital reference staff can 
use this as an opportunity for improving 
instruction on database use. An understanding 
of the user’s experience with other digital 
reference services may also help in improving 
the quality of service. 

When analyzing the results of the measure it 
may also prove useful to look for patterns in 
user behavior (e.g., a large percentage of 
users have tried Yahoo or Google, or many 
used the general periodical database, but seem 
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to be unaware of the specialized resources 
available). An awareness of these patterns 
may reflect a need to increase user awareness 
of both the type and quality of resources 
available both within the library and in 
general. 

The reference interview is the most logical 
and efficient vehicle for determining what 
other sources the user has tried. In some cases 
when gathering the data for this measure, it 
may be difficult to elicit the desired 
information from the patron in the reference 
interview process. The user may perceive the 
questions as intrusive and/or a waste of time, 
but if the need for this information can be 
expressed clearly, this may help the user to 
feel comfortable in relaying the information. 

It may also be useful to include in the 
reference interview, or on the reference 
question submission form, a question 
regarding how long the user has spent on the 
question, as this may give an indication of the 
user’s level of expertise or level of 
commitment to finding and answer. 

24. Reasons for use 

Definition: Reasons for use is a measure that 
reflects why the user chose to use the digital 
reference service either for the first time or as 
a repeat user. This measure can include such 
reasons as word-of-mouth, reputation of the 
library or traditional or digital reference 
service, advertising, reviews of service, 
distance from physical library facility, 
satisfaction with previous service, no other 
alternatives, etc. 

Rationale: Understanding the user’s reason 
for use can play a role in the continuing 
assessment and development of the marketing 
of the digital reference service and an analysis 
of the quality of the present service. This 
understanding can also help in knowing what 



the competition is or the availability of other 
resources to the target population. Knowing 
the reason for first time use can, in 
conjunction with other measures, assist in the 
analysis of the target and actual user 
population both for the library in general and 
the digital reference service in particular (e.g., 
physical distance from the library, hours are 
convenient, lack of time in busy life). This 
measure can also be an indicator of a 
successful program if the reason for first time 
use is positive reviews or endorsements in the 
media or by word of mouth, which can be 
used in requests for additional funding within 
the library or for the library as a whole. If the 
user is a repeat user, his or her reasons for 
repeat use can be helpful in determining the 
overall success, convenience, or importance 
of the digital reference service to the user. A 
close examination of the results of this 
measure is critical. 

Data Collection Procedure: 

Survey. 

Questionnaire. 

Interview. 

Collected by: Staff and Management or 
Supervisor. 

Frequency: Varies, can be conducted 

annually or on ongoing basis. 

Procedures: Determine how best to 

survey or interview representative sample 
of users. 

Develop survey, interview or 
questionnaire instrument, which may 
include questions related to other 
measures. 

Decide method for administering survey, 
questionnaire or interview. For example, it 
may be sent by mail or email, accessed 
electronically (via website or template 
survey sent out with each final response), 
or conducted by phone or in person. 
Determine return date for survey if 
necessary. 

Administer survey. 
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Tabulate and analyze results. 

Other measures to use in conjunction with 

this one: 

14 — Repeat Users. 

Helpful Tools - See Appendix I for: 

H. Sample Digital Reference Transaction 
Record. 

T. User Satisfaction Measures - Sample 
Survey/Questionnaire Questions. 

Issues and Considerations: Reasons for use 
by both first time and repeat users can vary 
widely so a close, objective, and detailed 
analysis of the questionnaires, survey or 
interview responses is important. The process 
can be time consuming, both in the 
administration and the subsequent evaluation 
of results, but the knowledge gained can be a 
strong indicator of the success of a digital 
reference service. 

It is simplistic to view repeat use solely as an 
indicator of success of the program or 
satisfaction of the user (though it can most 
certainly be those things). A close analysis of 
the data and narrative derived from this 
measure can indicate satisfaction or success, 
but also may indicate that users have no other 
alternative for a number of reasons, and even 
though they may be repeatedly using the 
service they may not be entirely satisfied. 

This measure is one of the most important to 
gather, and equally important is a thoughtful 
and objective assessment of the results. 

25. Reasons for non-use 

Definition: Reasons for non-use refers to an 
assessment of why people have never used the 
service or have used the service previously 
but are not using it currently. For people who 
have never used the service, this measure can 
include such reasons as: person was unaware 
of service; person is uncomfortable with 
technology; person does not have the 



technological resources to use the service; 
person believes they would prefer face-to-face 
service. For those who have used the service 
previously but do not use it currently, this 
measure can include such reasons as: user has 
been satisfied with previous use, but has no 
further need for service; user was dissatisfied 
with all or some of previous experiences with 
service; user has been able to meet subsequent 
information needs independently (self- 
service); user’s subsequent information needs 
have not met the policy parameters for use of 
the service. 

Rationale: Knowing why potential and 

previous users do not use the service is 
important in assessing the visibility of the 
service to the potential user population, 
understanding what any competition may be 
doing to attract users, understanding the 
sophistication level or ability (either 
cognitively or technically) of the potential 
user to access the service, and developing an 
awareness of any negative publicity regarding 
the service. Analyzing the reasons that people 
choose not to continue to use the service can 
be crucial in analyzing and correcting 
problems with the service if the reason for 
non-use after previous service is 
dissatisfaction with the service for some 
reason. On the other hand, this measure can 
also be an indicator of satisfaction with the 
service (i.e., the user has had his/her 
information need' met) or a reflection of the 
fact that the user has had no subsequent need 
of the service. Non-use should not 
automatically be viewed as a negative. 

Data Collection Procedure: 

Survey. 

Questionnaire. 

Interview. 

Collected by: Staff and Management or 

Supervisor. 

Frequency: Varies, can be conducted 

annually or on ongoing basis. 
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Procedures: Determine how best to 

survey or interview representative sample 
of users. 

Develop survey, interview or 
questionnaire instrument, which may 
include questions related to other 
measures. 

Decide method for administering survey, 
questionnaire or interview. For example, it 
may be sent by mail or email, accessed 
electronically (via website or template 
survey sent out with each final response), 
or conducted by phone or in person. 
Determine return date for survey if 
necessary. 

Administer survey. 

Tabulate and analyze results. 

Helpful Tools - See Appendix I for: 

T. User Satisfaction Measures — Sample 
Survey/Questionnaire Questions. 

Issues and Considerations: It may be 
difficult to define the target population when 
measuring reason for non-use by those who 
have never used the service before. 

It has always been difficult in assessment 
measures to reach the non-user in order to 
determine why they do not use a particular 
service. This is also true in the assessment of 
digital reference services and may be even 
more so because in some cases (i.e., national 
or global) there is no clearly defined service 
population. 

In cases where there is a defined service 
population, collaboration with other 
departments or agencies (other administrative 
or academic departments in the case of 
universities, or other government or 
municipal agencies in the case of public 
libraries) may provide an additional 
mechanism for reaching the non-user. 



It is important that the reasons for non-use be 
analyzed in a careful, objective, and 
thoughtful manner. One should not assume 
that a lack of use is an indicator of 
dissatisfaction with the service, poor 
marketing or lack of accessibility (though 
these cam certainly be valid reasons). A close 
look at the reasons for non-use, while a 
somewhat complex procedure, will help to 
assess the user’s the relative satisfaction or 
dissatisfaction with the service. 

Some field testers indicated that this measure 
may not be particularly useful on a regular 
basis because a service already may have 
more users them cm be handled. 

26. Improvements needed and 
additional services that need to 
be offered. 

Definition: Improvements needed describes 
those areas in which the service cm benefit 
from positive chmge as determined by user 
suggestions md complaints, as well as by 
digital reference staff assessment of the 
service. Included in this measure is a 
determination of possible additions to service. 

Rationale: A determination of both user md 
staff suggestions for improvement of md 
additions to existing service is essential to m 
malysis of the quality of the existing digital 
reference service. Improvements needed may 
include faster turnaround time or better 
communication on the part of the staff. 
Additional services that need to be offered 
may include additional hours or days, increase 
in the types of reference questions mswered, 
adaptive technology for users with special 
needs, etc. 

Data Collection Procedure: 

Survey. 

Questionnaire. 

Interviews/focus groups. 
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Other: Unsolicited comments from users and 
staff. 

Collected by: Staff and Management or 
Supervisor. 

Frequency: Varies, can be conducted 

annually or on ongoing basis. 

Procedures: Determine how best to 

survey or interview a representative 
sample of users and staff. If conducting a 
focus group, determine the participants in 
the focus group and contact them. 

Develop survey, interview or 
questionnaire instrument, which may 
include questions related to other 
measures. 

Decide method for administering survey, 
questionnaire or interview. For example, it 
may be sent by mail or email, accessed 
electronically (via website or template 
survey sent out with each final response), 
or conducted by phone or in person. 
Determine return date for survey if 
necessary. 

Administer survey or interview or conduct 
focus group meeting. 

Tabulate and analyze results. 

Both user and staff input should be 
gathered for this measure. In many cases, 
the users may offer unsolicited comments 
regarding what they would like to see 
improved or added. Keeping track of these 
unsolicited remarks should be undertaken 
if not done currently and these remarks 
should be incorporated with the 
recommendations gathered from users 
through the more formal methods. 

Focus groups may be more practical for 
staff than users. In addition, inclusion of 
this topic for discussion at regularly 
scheduled staff meetings provides for 
ready-made collection of the information 
needed for this measure. If conducting a 
focus group for staff, the facilitator must 



be chosen well so that the staff sees that 
person as objective. 

Helpful Tools - See Appendix I for: 

T. User Satisfaction Measures - Sample 
Survey/Questionnaire Questions. 

Issues and Considerations: Suggestions for 
improvement may come from either the staff 
or the user and may reflect differing 
expectations or recommendations. A detailed 
and objective analysis of the 
recommendations for improvement may help 
to define and/or expand the service. The 
suggestions for improvement may be 
perceived as, or in fact may be, intended as 
criticism, and looking at the results derived 
from this measure may create uneasy or 
uncomfortable situations among staff or 
between staff and supervisors. In addition, it 
is important to distinguish between those 
suggestions for improvement or additions to 
services that can be feasibly implemented, 
and those that may not fit within the goals and 
mission of the service that or may not be 
possible due to budgetary or staffing 
concerns. For example, a reasonable 
improvement needed may be an improvement 
in average turnaround time from three days to 
two, while an unfeasible suggestion for 
improvement may be having someone staff 
the reference desk twenty four hours a day, 
seven days a week. Feasible additions to 
service may include staffing the digital 
reference desk on Sunday evenings, while 
adding a complex interactive video or voice 
component to the reference service may be 
outside of the budget capabilities of the 
library. 

27. Satisfaction with staff service 

Definition: Satisfaction with staff service 
measures the user’s perception about how 
well the digital reference service staff met his 
or her information needs. Satisfaction with 
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staff service can include the following 
components: 

• Perceived personalized service reflects 
how individualized the service given by 
the staff to the user was, and can include 
such factors as responding to the user by 
name, addressing specialized requests and 
needs, and responding with customized 
answers to meet the user’s needs rather 
than with template type responses. 

• Perceived demeanor and attitude of 
staff in communication reflects such 
qualities as the perceived eagerness of the 
staff to be helpful, the tone of the 
language used in communication, 
sensitivity to privacy issues during the 
reference interview process and other 
parts of the transaction, and willingness to 
“listen.” Satisfaction with staff can also 
include the user’s perception of the 
demeanor and attitude of any support and 
technical staff the user may have occasion 
to contact. 

• Perceived timeliness of response reflects 
whether the user feels that his or her 
reference question was answered in an 
acceptable period of time and can include 
the responsiveness to “rush” questions, 
whether the question was answered more 
quickly than was anticipated by the user, 
or whether or not the question was 
answered in the expected time frame. The 
perceived timeliness of response can also 
include the perception of the timeliness of 
initial acknowledgement to the user of 
receipt of the reference question by the 
reference staff. 

Rationale: Generally, this measure can be 
useful in scheduling of staff, in assessing the 
communication skills of the staff, and in 
assessing the quality of service based on the 
users’ perceptions their interactions with staff. 

Specifically, this measure will be useful in 
developing staff skills needed for providing a 



measure of personalization in a technical 
environment in which the user is removed 
from the staff member assisting them. It will 
also be helpful in developing a model and 
procedures for interacting with the user. For 
example, it may be decided that any email 
correspondence with the user will include a 
form of address that includes the user’s name 
(e.g., Dear Ruth). In many cases, user 
satisfaction with a reference service 
(traditional or digital) depends heavily on the 
perceived personalization of service and 
positive attitude and demeanor regardless of 
other factors such as correctness of answer or 
speed of response. 

This measure is also especially important in a 
digital reference setting because the 
traditional measures of attitude and demeanor, 
such as body language, facial expression, tone 
of voice, etc., will not be apparent to the user 
(unless the transaction is a real time video or 
voice transmission). 

An understanding of the user’s perception of 
the timeliness of response can be used to 
develop turnaround time policies for the 
service. Perceived timeliness of response can 
also assist in the scheduling of staff for the 
service. This will be especially useful if it can 
be ascertained when submission of rush 
questions will be at a peak (the end of the 
semester or during midterms in an academic 
setting, in the appropriate season for public 
libraries located in resort areas affected by 
seasonal influx of population, etc.). 

Data Collection Procedure: 

Survey. 

Interview. 

Collected by: Staff and Management or 

Supervisor. 

Frequency: Varies, can be conducted 

annually or on ongoing basis. 
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Procedures: Determine how best to 

survey or interview representative sample 
of users. 

Develop survey, interview or 
questionnaire instrument, which may 
include questions related to other 
measures. 

Decide method for administering survey, 
questionnaire or interview. For example, it 
may be sent by mail or email, accessed 
electronically (via website or template 
survey sent out with each final response), 
or conducted by phone or in person. 
Determine return date for survey if 
necessary. 

Administer survey. 

Tabulate and analyze results. 

Helpful Tools - See Appendix I for: 

T. User Satisfaction Measures - Sample 
Survey/Questionnaire Questions. 

Issues and Considerations: This measure is 
heavily dependent upon the subjective 
assessment of the user on a subjective aspect 
of the digital reference service, but it should 
not be underestimated in its contribution to 
the perceived quality of a digital reference 
service. Personalized service, positive attitude 
and demeanor, and timeliness of response in 
any service industry are often seen as the 
most important measures of quality. 

The personalization of digital reference 
service may be especially critical, and special 
attention may need to be paid to this area, 
because in many cases the very nature of 
digital communication is perceived to be 
impersonal. Except in the cases of interactive 
video or voice-based digital reference service, 
all communication takes place via the 
transmission of typed text to and from the 
user and staff member. Strategies for 
personalization of service that are available to 
the traditional reference service provider (i.e. 
greeting the patron first, making direct eye 



contact, open body language, remembering a 
patron’s name on repeat uses of the service, 
providing specialized verbal commentary on 
the resources provided) are not possible and 
other means of approximating these methods 
may need to be devised. 

Some of the voice sync services (e.g., Live 
Person, Human Click) provide a minimal 
“personalized” greeting. It would be 
interesting to learn if this automated attempt 
at personalization is perceived as such by the 
user. 

It can be difficult to convey a positive, 
proactive demeanor and attitude in the digital 
reference environment because again the 
normal indicators of attitude and demeanor in 
face-to-face interactions are not possible in a 
digital reference setting (unless interactive 
video is being utilized). A user may submit a 
reference question to the service and if receipt 
of the question is not acknowledged 
immediately, this may be interpreted 
(correctly or incorrectly) as a poor attitude or 
demeanor. Also, as has been documented, 
communication via email has been shown to 
affect a person’s perceived demeanor. The 
staff member may be warm and open and 
positive in person, but the communication via 
email may easily be perceived as neutral or 
non-caring based on the language being used. 
An analysis of the data gathered in this 
measure may indicate a need for written 
communication training, especially in the 
arena of digital communication. Formal 
methodologies and procedures for conveying 
a positive attitude may be warranted (such as 
acknowledgement of receipt of a question as 
soon as possible after such receipt even if the 
question cannot be addressed immediately). 
Use of lively positive language can also be 
used to convey attitude and demeanor. 

Perceived timeliness of response may reveal a 
disparity between what the service sets as a 
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reasonable expectation for response time and 
the expectations of the user. Factors that are 
entirely dependent on the user’s situation may 
affect their expectation of turnaround time for 
the answer to their question. These can 
include: rush questions, underestimation of 
the complexity of their question, incorrect or 
unreasonable expectations regarding 
availability or format of resources that may be 
used to answer the question. Factors over 
which the digital reference service provider 
has some control include the allocation of 
staff to the digital reference service, 
scheduling of staff during peak usage times 
when it is known that rush questions may be 
asked, expertise of staff in answering specific 
types of questions, and knowledge of staff 
regarding available resources. Use of the data 
derived from this measure can help in the 
determination of a reasonable turnaround 
time, which should be posted with the digital 
reference service policies. 

Questions related to this measure can be 
included with other user satisfaction measure 
questions in an interview or survey. Ideally, 
this interview or survey should be conducted 
at the end of the reference transaction process 
because it is sometimes difficult to track 
down users once the transaction has been 
completed. 

28. Delivery mode satisfaction 

Definition: Delivery mode satisfaction is a 
measure of how satisfied the user is with the 
electronic delivery method used to answer the 
reference query. This measure should reflect 
user satisfaction with answers sent via email 
or chat, or pushing of websites or downloads 
to the user, and should reflect relative 
satisfaction if possible with the digital 
delivery mode as compared to traditional 
delivery mechanisms (ILL, delivery of books, 
self service copying). 



Rationale: An understanding of the user’s 
satisfaction with the method of delivery for 
the answer provides useful information in the 
ongoing development of the service and 
decisions regarding possible additions to 
delivery methods (e.g., adding chat, voice or 
video technology). This measure will also 
provide information regarding the technology 
available to the user (perhaps they do not 
have the knowledge, ability, or desire to 
participate in a chat or video transaction) or 
are still using email systems that do not allow 
for the receipt of attachments. This 
understanding can also be used to develop 
collaborative arrangements with other 
departments within the library or other 
external agencies. For example, if a book in 
the library collection provides the best answer 
to a question, a collaboration between the 
lending department and the digital reference 
service could allow for the placing of a 
request for the user without having the user 
initiate the request with that department. 
Finally, this measure can be used to address 
the need for training and equipment updates. 

Data Collection Procedure: 

Survey. 

Interview. 

Collected by: Staff and Management or 
Supervisor. 

Frequency: Varies, can be conducted 

annually or on ongoing basis. 

Procedures: Determine how best to 

survey or interview representative sample 
of users. 

Develop survey or interview questionnaire 
instrument, which may include questions 
related to other measures. 

Decide method for administering survey, 
questionnaire or interview. For example, it 
may be sent by mail or email, accessed 
electronically (via website or template 
survey sent out with each final response), 
or conducted by phone or in person. 
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Determine return date for survey if 

necessary. 

Administer survey. 

Tabulate and analyze results. 

Helpful Tools - See Appendix I for: 

T. User Satisfaction Measures - Sample 
Survey/Questionnaire Questions. 

Issues and Considerations: While this 

measure may provide data that can be used in 
consideration of enhancements to the service 
(by possibly adding desired delivery 
methods), it may also be decided that such 
desired changes or enhancements to the 
service may not be possible due to budgetary 
or technological constraints. 

Questions related to this measure can be 
included with other user satisfaction measure 
questions in an interview or survey. Ideally, 
this interview or survey should be conducted 
at the end of the reference transaction process 
because it is sometimes difficult to track 
down users once the transaction has been 
completed. 

29. Impact of service on user 

Definition: Impact of service on user is a 
critical outcome measure that describes how 
the user has been affected by the service. 
Impact on the user will include such effects 
as: solving an information problem, 

answering an information need, saving time, 
and access to resources the user might not 
otherwise have. It will also include aspects of 
the patron’s life that have been impacted by 
the service (e.g., user finds employment as a 
result of information provided about the 
prospective company prior to the employment 
interview; user is able to participate in a 
clinical trial for a serious illness; user is able 
to find the address of a company to file a 
customer complaint; student is able to locate 
six periodical sources for social studies 



project; college student is able to graduate 
because she completed her Master’s Thesis; 
medical or nursing students or physicians are 
able to treat a patient successfully due to 
information received). 

Rationale: Determination of the impact of the 
digital reference service on the user is 
extremely important as a validation that the 
service is an important part of the library 
service and is important to the user. This 
measure is especially useful in gamering 
political and/or financial support for the 
service itself or the library as a whole. This 
measure can also provide a sense of personal 
satisfaction to individual staff members who 
are necessarily removed from the user 
physically in a digital reference environment 
and otherwise may not learn the results of 
their efforts. The effect of understanding the 
impact of the service on the user can improve 
employee morale and job satisfaction. 

Data Collection Procedure: 

Survey. 

Interview/Focus Groups. 

Other: Unsolicited input from user. 

Collected by: Staff and Management or 
Supervisor. 

Frequency: Varies, can be conducted 

annually or on ongoing basis. 

Procedures: Determine how best to 

survey or interview representative sample 
of users. 

Develop survey, interview or 
questionnaire instrument, which may 
- include questions related to other 
measures. 

Decide method for administering survey, 
questionnaire or interview. For example, it 
may be sent by mail or email, accessed 
electronically (via website or template 
survey sent out with each final response), 
or conducted by phone or in person. 
Determine return date for survey if 
necessary. 
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Administer survey. 

Tabulate and analyze results. 

One suggestion for getting at some of this 
information is to draw a random sample of 
users who received answers. After a 
period of time (typically two to six 
months) contact the user to determine 
impact. This process can be facilitated at 
the time of the digital reference 
transaction to see if the user is willing to 
be contacted at a later date to be 
interviewed or to fill out a survey 
regarding the impact of the service. 

Unsolicited stories or anecdotes about 
impact are often provided by users. These 
should be collected and included with 
those gathered by more formal methods 
when doing the assessment. 

Helpful Tools - See Appendix I for: 

T. User Satisfaction Measures - Sample 
Survey/Questionnaire Questions. 

Issues and Considerations: Impact of the 
service on the user is arguably the most 
important measure of digital reference service 
with regard to quality of service. Anecdotal 
and qualitative evidence of a digital reference 
service’s impact is extremely compelling for 
political and funding reasons. Impact on the 
user’s life is often one topic that library or 
organizational boards and government 
officials and legislators ask about when 
considering funding for libraries in general 
and digital reference service in particular. 
Unfortunately, this measure is also one of the 
most notoriously difficult to determine. In 
many cases the full impact of the service on 
the user may not be apparent to the user until 
long after the question has been answered. It 
may be difficult to track down the user for 
participation in a survey or intend ew/focus 
group. Some examples of impact on the user 



that may not occur until relatively later 
include the following scenarios: 

• The user is a graduate student who needs 
to locate an obscure article for use in the 
development of her thesis. It may take 
several semesters for the thesis to be 
completed. 

• The user is a multiple myeloma patient 
who requests information on clinical trials 
and bone marrow transplants. He 
subsequently undergoes the transplant and 
his life has been extended by several years 
as a result. 

• The user requests information on how to 
write a resume and then is offered a 
position by several companies to which 
she applied. 

In other cases the impact of the service may 
be real, but not recognized by the user (as in 
many other situations, good library service is 
often taken for granted by the user). 

If the digital reference service is able to 
collect anecdotal, qualitative, or statistical 
data on impact of the service to the user 
(either solicited or unsolicited), the data can 
be used to advertise the service, to justify 
expenditures on the service and in lobbying 
efforts for funding. 

30. User demographic data 

Definition: User demographic data is data 
that defines characteristics of the user 
population and includes such attributes as age, 
race, gender, education, employment and 
profession. For academic library services, 
such user categories as faculty, staff, and 
student (with further classification by 
undergraduate, graduate, doctoral or 
professional) should be included. For public 
library service such user categories as age 
(with further classification by child, young 
adult or adult), occupation, and residency 
status in legal service area might be collected. 
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Rationale: An analysis of the users’ 

demographic data can determine if specific 
characteristics of the users fall into detectable 
patterns that can be used to modify or 
enhance the service. Additionally, this data 
can be used for collection development to 
determine the range of resources that might be 
warranted to meet the specific needs of a 
particular demographic group. Some 
examples include: online children’s 

encyclopedias for users under the age of 10, 
online peer reviewed e-joumals for graduate 
students and faculty, online databases in 
Spanish (both in the search interface and the 
content) for a largely Hispanic population, 
self-help and support materials for young 
mothers). Additionally, this demographic data 
can be used to determine which demographic 
users groups are not utilizing the service, and 
efforts can be made to determine why and to 
address the unmet needs of those groups. 

Data Collection Procedure: 

Survey. 

Other: Analysis of user data provided by 
various sources such as registration files, 
digital reference submission forms, etc. 

Collected by: Staff and Management or 
Supervisor. 

Frequency: Varies, can be conducted 

annually or on ongoing basis. . 

Procedures: Determine how best to 

survey or interview representative sample 
of users. 

Develop survey, interview or 
questionnaire instrument, which may 
include questions related to other 
measures. 

Decide method for administering survey, 
questionnaire or interview. For example, it 
may be sent by mail or email, accessed 
electronically (via website or template 
survey sent out with each final response), 
or conducted by phone or in person. 
Determine return date for survey if 
necessary. 



Administer survey. 

Tabulate and analyze results. 

Helpful Tools - See Appendix I for: 

H. Sample Digital Reference Transaction 
Record. 

T. User Satisfaction Measures — Sample 
Survey/Questionnaire Questions. 

Issues and Considerations: The collection of 
user demographic data directly from the user 
may be perceived as intrusive by the user. If 
the data is to be collected from other sources, 
it is important to focus on those demographic 
categories that will prove the most useful in 
improving quality of service. In the interest of 
saving time, it is important not to collect 
demographic data, just for the sake of 
collecting it. Often such data must be 
gathered from a variety of sources and this 
can be time consuming. In the case of 
gathering demographic data concerning 
children, only non-identifying data must be 
collected. Also, if the data is to be collected 
directly from the child via a survey, parental 
permission may be needed before the child 
can complete the survey. 
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COST 

31. Cost of digital reference service 

Definition: The cost of digital reference 
service is the total cost for providing the 
service and includes the cost of staff 
(including salaries and benefits), digital 
resources (licensing fees, support, etc.), 
infrastructure (including hardware, software, 
telecommunications, website hosting fees, 
domain name fees), lease, mortgage or rental 
costs, marketing and advertising, training, 
print materials (if used for digital reference), 
supplies, and any other costs associated with 
the digital reference service. In cases where 
digital reference is provided as an integrated 
component of the regular reference service or 
where staff and resources are shared between 
the digital reference service and the regular 
reference services, costs should be prorated. 

Rationale: Knowing the cost of any library 
service is crucial when developing budgets 
and in making managerial decisions regarding 
allocation of resources. The cost of providing 
digital reference service is important at this 
overall management level and also for 
planning and allocation of resources within 
the service. 

Data Collection Procedure: Other: 

Collection and analysis of cost component 
data including invoices, personnel 
expenditures, advertising expenditures, 
tuition/education costs, etc. 

Collected by: Manager or Supervisor. 
Frequency: Varies but typically 

conducted annually. 

Procedures: Gather required infor- 

mation from annual reports, personnel 
records (for salary information), and/or 
the business office. 

Helpful Tools - See Appendix I for: 

U. Cost Worksheet. 



V. Cost - Electronic Resources Prorate 
Worksheet. 

Issues and Considerations: It has always 
been difficult to determine exact costs of any 
service within a library for many reasons. In 
many cases staff and resources are utilized by 
more than one service area within the library, 
and it is difficult to prorate out costs for any 
one area. Some resources are utilized both 
within the library and externally (as in the 
case of remote access to databases), so it is 
difficult to ascribe the cost to any one 
department. These same issues also affect the 
digital reference service. Some of the most 
costly resources for the provision of digital 
reference are subscriptions and licenses to 
online resources and databases. These 
resources are also available for use by other 
departments and by the patron from both 
within the library and at home. Also, different 
vendors have been varyingly successful or 
interested in providing meaningful statistics 
and data about database use. In many cases it 
is impossible to determine what percentage of 
costs can be allocated to the digital reference 
service (especially when authentication is by 
IP address only). Staff members perform the 
duties of traditional and digital reference at 
the same time, and keeping track of time 
allocated to either can be problematic. It is 
important, however, to make an attempt to 
determine costs. 

32. Cost of digital reference service 
as a percent of total reference 
budget 

Definition: Cost of digital reference as a 
percentage of total reference budget is a 
comparison of the total cost of the digital 
reference service in relation to the total 
reference budget, which includes traditional, 
hybrid, and digital reference service. This 
comparison is expressed as a percentage. 



Cost of digital reference X 100 
Total reference budget 
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Rationale: It is important to be able to know 
the relative cost of the digital reference 
service to the total reference budget so 
appropriate allocation can be made to the 
digital reference service (and also to the 
traditional service, if warranted). Budgetary 
considerations (regarding the relative cost to 
relative benefit) are important in any 
organizational setting, and are critical in the 
digital reference setting because the cost of 
providing digital reference can often be 
comparatively high due to possible staffing 
issues (both in terms of expertise and time), 
cost of resources (software, online databases), 
training needs (staff must be trained not only 
in traditional reference service but also in 
technical and computer service), etc. 

Data Collection Procedure: Other: 

Collection and analysis of cost component 
data including invoices, personnel 
expenditures, advertising expenditures, 
tuition/education costs, etc. 

Collected by: Manager or Supervisor. 
Frequency: Varies but typically 

conducted annually. 

Procedures: Gather required infor- 

mation from annual reports, personnel 
records (for salary information), and/or 
the business office. 

Determine the cost of the digital reference 
service and then divide by the total 
reference budget. Multiply by 100. 

Helpful Tools - See Appendix I for: 

W. Cost of Digital Reference as Percent 
of Total Reference / Total Library 
Budget. 

Issues and Considerations: One of the most 
critical difficulties in determining this 
extremely important measure is that it has 
been difficult for libraries to determine the 
real costs of the overall reference budget. 
Confusion and disagreement about line item 
allocations (especially with regard to the 



purchase of online databases and prorating of 
personnel cost for those staff members who 
perform multiple job duties) may make it 
difficult to obtain a point of comparison. 
Additionally, in the overall library budget, 
automation and reference budget lines may be 
separate even though the cost of digital 
reference service may not make such 
distinctions. Prorating the cost of fee-based 
online databases (which is often a substantial 
portion of a library’s budget) may also prove 
problematic in that it is often difficult to 
determine use strictly by the digital reference 
service versus use for regular reference 
service, or use by the patrons in the library 
itself versus by remote users. 

This measure will also be more difficult when 
digital reference service is being provided as 
part of a collaborative effort in a consortium. 

33. Cost of digital reference as a 
percent of total library or 
organizational budget 

Definition: Cost of digital reference service 
as a percentage of total library or 
organizational budget is a comparison of the 
cost of the digital reference service to the total 
library or organizational budget expressed as 
a percentage. 



Cost of digital reference X 100 

Total organizational budget 



Rationale: In any library dr organizational 
setting, it is crucial to understand the relative 
cost of any service that is part of the 
organization to the organization as a whole. 
Informed budgeting decisions and allocation 
of resources throughout the organization can 
only be made if the cost of each service is 
known. Additionally, the cost of the digital 
reference service in comparison with the total 
organizational budget can be used to leverage 
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for more resources if the relative cost is much which exist solely to provide digital 
less than the benefit and use generated by the reference), 
digital reference service. 

Data Collection Procedure: Other: 

Collection and analysis of cost component 
data including invoices, personnel 
expenditures, advertising expenditures, 
tuition/education costs, etc. 

Collected by: Manager or Supervisor. 

Frequency: Varies but mainly 

annually. 

Procedures: Gather required infor- 

mation from annual reports, personnel 
records (for salary information) and/or the 
business office. 

Determine the cost of the digital reference 
service and then divide by the total library 
or organizational budget. Multiply by 1 00. 

Helpful Tools - See Appendix I for: 

W. Cost of Digital Reference as Percent 
of Total Reference / Total Library 
Budget 

Issues and Considerations: It is often 
difficult to determine an exact cost for digital 
reference service in settings where staff 
members may be shared between a formal 
digital reference service and traditional 
reference service, or where the digital 
reference service is an integrated part of 
traditinal reference service. It may also be 
difficult to determine costs when online 
databases are used by reference staff, other 
library staff and patrons alike, both in the 
physical library and from remote settings. 

Additional problems may include prorating 
costs for equipment and software that may be 
used by the digital reference staff and other 
library staff. 

This measure will not be as useful for those 
digital reference services that do not exist 
within the context of a larger library service 
(e.g. AskERIC and other AskA services, 
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STAFF TIME EXPENDED 

34. Percent of staff time spent 
overseeing technology 

Definition: Percent of staff time overseeing 
technology is the percentage of time that staff 
spend installing, repairing, and maintaining 
hardware and software compared to total staff 
time devoted to digital reference service (staff 
time overseeing technology plus other staff 
time devoted to digital reference service). 



Staff time overseeing technology X 100 

Total staff time for digital reference 



Rationale: The nature of the provision of 
reference service is changing both in the 
traditional setting of the library and in the 
digital reference service arena. In both cases it 
has been reported, anecdotally and in studies, 
that reference staff spend an increasing 
amount of time overseeing the technology. In 
digital reference, such time can be expended 
in activities such as creating, maintaining, and 
updating the digital reference website, 
downloading updates for virus protection, and 
diagnosing and correcting problems with 
hardware, software, and peripherals (modems, 
scanners, printers, monitors, configurations, 
etc.). The skills required to perform these 
tasks are in addition to those required for 
providing quality reference service. It is 
important for management to know what 
percentage of time their staff is devoting to 
each of these dual roles. This information can 
be helpful in making hiring decisions, staffing 
and compensation decisions, and when 
analyzing and evaluating employee 
performance and job satisfaction. 

Data Collection Procedure: 

Sampling. 

Survey. 

Interview. 



Other: analysis of manual log. 

Collected by: Manager or Supervisor or 
designated staff member. 

Frequency: Logs are kept daily. It is 

recommended that this detailed 
information be gathered over the course of 
a typical week and repeated several times 
throughout the year. 

Survey/interview is administered to staff 
on periodic basis such as once per year. 
Procedures: Determine which activities 

meet the criteria of overseeing/ supporting 
technology. 

Develop a manual log for registering this 
use of time. 

Develop and administer a 
survey/interview instrument that elicits 
staff perceptions of the amount of time 
spent overseeing the technology. 

For this measure, count all time creating, 
maintaining, and updating the digital 
reference website, downloading updates 
for virus protection, and diagnosing and 
correcting problems with hardware, 
software and peripherals (such as 
modems, scanners, printers, monitors, 
configurations). Also include staff time 
spent on learning how to oversee the 
technology as delineated above. 

I'n many cases, staff members are 
responsible for overseeing technology that 
is used by several departments within a 
library, including the digital reference 
service. It may be difficult to break out the 
portion of their time that should be 
allocated to the digital reference service. 
Some options for dealing with difficult 
problems include: 

• Using the same percentage derived 
from Measure 33 — Cost of Digital 
Reference Service as a Percent of 
Total Library or Organizational 
Budget. This is the easiest but also the 
least precise method. It is artificial and 
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does not get at the very real fact that 
time spent overseeing technology 
related to digital reference may not be 
directly proportional to the percentage 
of the total budget that the digital 
reference encompasses. 

• Only measure the time spent on 
overseeing technology that directly 
relates to the digital reference service. 

• Only include time spent on overseeing 
technology by staff members who are 
directly involved in the digital 
reference service. 

Helpful Tools — See Appendix I for: 

X. Staff Time Expended - Percent of 
Time Overseeing Technology. 

Issues and Considerations: Collecting the 
detailed information required by this measure 
can be time consuming and confusing, and 
will require detailed record keeping by the 
digital reference staff. In spite of the 
difficulties in gathering and calculating the 
data necessary for this measure, it is one of 
the most important measures from a 
management perspective. In addition, the 
tasks and skills required to oversee the 
technology may not match those required to 
provide direct digital reference service. 
However, due to budgetary concerns or to the 
fact that the service may have evolved in an 
unplanned way, many digital reference 
librarians also function as their own technical 
support. 

35. Percent of staff time spent 
assisting users with technology 

Definition: Percent of staff time spent 
assisting users with technology is the percent 
of staff time spent in informal, non-scheduled, 
one-on-one technology training with users 
compared to total staff time devoted to digital 
reference. This measure refers only to 
assistance offered to those users using the 



digital reference service. It does not refer to 
assisting those users who are utilizing the 
traditional reference service. 



Staff time assisting users with technology X 100 
Total staff time for digital reference 



Rationale: The results of the initial site visits 
for this study and others have suggested, 
anecdotally at least, that there appears to be a 
significant percent of staff time assisting users 
with the technology in addition to the 
provision of actual reference service. Such 
activities include teaching users how to: 
download software (such as Adobe Acrobat 
Reader) for viewing pdf files; save images to 
disk; import results from a periodical database 
into a spread sheet or bibliography software, 
etc. These types of duties require special skill 
on the part of staff. An understanding of the 
percent of time staff spend on such tasks can 
assist management in making staff 
assignments, developing training programs 
for staff, hiring decisions. They can also be 
crucial in assessing staff performance and job 
satisfaction. 

Data Collection Procedure: 

Sampling. 

Survey. 

Interview. 

Other: analysis of manual log. 

Collected by: Manager or Supervisor or 
designated staff member. 

Frequency: Logs are kept daily. It is 

recommended that this detailed 
information be gathered over the course of 
a typical week and repeated several times 
throughout the year. 

Survey/interview is administered to staff 
on periodic basis such as once per year. 
Procedures: Determine which activities 

meet the criteria of supporting users with 
technology. 
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Develop a manual log for registering this 
use of time. 

Develop and administer a survey/ 
interview instrument that elicits staff 
perceptions of the amount of time spent 
assisting users with technology. 

Helpful Tools - See Appendix I for: 

Y. Staff Time Expended - Percent of 
Time Assisting Users with 
Technology. 

Issues and Considerations: The nature of 
reference service and the skills needed to 
provide reference service are rapidly 
changing. As a result of the increased role 
technology plays in information provision, the 
role of the reference librarian has been 
steadily shifting to encompass both the 
provision of information and the provision of 
instruction to the user. While reference 
librarians have always had to provide some 
instruction to the user (how to use 
complicated indexes, how to use the subject 
headings indexes, how to read a citation, how 
to use the card catalog, how to properly cite 
sources), the type of instruction required now 
includes providing instruction to the user in 
the sphere of technological and computer- 
based knowledge. For example, reference 
librarians are now expected to instruct users 
in how to: navigate a website, download 
documents, use Boolean logic in online 
databases and Internet search tools, handle 
printer problems, etc. The collection of data 
for this measure will require development and 
maintenance of logs detailing the amount of 
t im e the digital reference staff spend assisting 
users with the technology. This measure, like 
Measure 35 — Percent of Staff Time 
Overseeing the Technology is important in 
hiring and assignment of staff for digital 
reference service and in the development of 
training for staff. Thus, even though this 
measure is complicated and time consuming, 
it is important to get an accurate measurement 



regarding the percentage of time digital 
reference staff spend instructing users in the 
use of technology. 
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OTHER ASSESSMENT 
OPTIONS 

The following three techniques can also be 
incorporated in the assessment process and 
can be used for quality control and training. 

Peer Review 

Peer review is a process where reference 
librarians review and critique the email 
question- answer sets and/or the transcripts 
from chat reference sessions performed by 
their peers, and provide each other with 
feedback and, where applicable, suggestions 
for improvement. Review needs to be based 
on established and formally stated criteria for 
reference performance and may include 
evaluation of appropriateness of response, use 
of standard scripts, accuracy of answer, 
completeness of answer, use of resources, 
referral and appropriateness of referral. It is 
not recommended that this process be used as 
input for individual performance appraisal 
tied to raises or promotion. 

Enhanced Reference Logs 

Enhanced reference logs are records of digital 
reference experiences kept by individual 
librarians to document professional insights, 
observations, issues, difficulties, successes, 
etc., that they experience in providing digital 
reference. These logs can be especially useful 
for capturing issues/events that were not 
anticipated or planned for when the service 
was designed and for collecting professional 
input on how the service, professional 
training, and library policies and procedures 
relating to digital reference can be improved. 

Librarian Discussion Groups 

Library discussion groups are meetings that 
provide a forum in which digital librarians 
can discuss their work and share their 
experience, knowledge of resources, and 
suggestions for improving digital reference 



services, find help with difficult questions or 
situations, and generally support each other in 
their professional growth. 
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QUALITY STANDARDS 

Developing Quality Standards 

This section of the manual provides guidance 
on the development and use of quality 
standards related to digital reference services. 
The actual level or standard included in a 
quality standard is determined by library staff 
and administration, but the need for and use 
of standards, example quality standards, and 
the process for developing a quality standard 
can be described here. 

Definition and Importance 

A quality standard is a specific statement 
of the desired, or expected level of 
performance that should be provided by a 
service or some aspect of that service. A 
quality standard can be measured to 
determine the degree to which that standard 
is, in fact, being met. A quality standard 
defines the level of performance that an 
organization is willing to accept for a 
particular service or activity. Quality 
standards are important because they: 

• encourage library staff and administration 
to discuss and agree upon what constitutes 
“quality” for a specific service; 

• provide clear guidance as to the expected 
quality that a particular service or activity 
should offer; 

• educate staff - and especially new staff - 
as to the expected quality of service that 
should be provided; 

• recognize that there maybe differing 
acceptable levels of quality for different 
aspects of digital reference services; and 

• provide a basis for rewards and 
demonstrating/reporting accountability. 

Quality standards draw upon statistics and 
measures that are described elsewhere in this 
manual. Quality standards are not 
performance measures. A performance 
measure might be “correct answer fill rate” 



whereas the quality standard might be “the 
digital reference service will have a correct 
answer fill rate of 65%.” 

Example Quality Standards 

We propose the following quality 
standards to be used with the provision of 
digital reference services. 

1. Courtesy. Users of the digital reference 
service will rate the courtesy of library 
staff providing digital service as a score of 
X on a scale of 1 (discourteous) to 7 (very 
courteous). The process for collecting data 
to produce this standard is a survey that 
can be attached to the digital reference 
response. Or, users of the digital reference 
service may be contacted at a later date 
via telephone, email, or other means to 
obtain their assessment. 

2. Accuracy. Library staff will provide a 
correct answer fill rate of XX%. Correct 
answer fill rate is the percentage of correct 
answers given to all questions received. 
The process for producing data for this 
standard is that a peer group of digital 
reference staff will examine a sample of 
digital reference transactions logs and 
judge the answers provided to be: correct, 
incorrect, referred, or cannot determine. 
The percentage of those responses judged 
to be “correct” will be compared against 
all questions asked to determine the 
accuracy of the digital reference services 
being provided. 

3. Satisfaction. Users of the digital 
reference service will rate their overall 
satisfaction with the service as a score of 
X on a scale of 1 (very dissatisfied) to 7 
(very satisfied). The process for collecting 
data to produce this standard is a survey 
that can be attached to the digital 
reference response. Or, users of the digital 
reference service may be contacted at a 
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later date via telephone, email, or other 
means to obtain their assessment. 

4. Repeat Users. At least XX% of all users 
of the digital reference service will ask 
two or more questions per week, month, 
or other time period. The process for 
collecting this data is to analyze the log 
files and determine how many times the 
same IP address, user ID, or other 
identifier has asked a question of the 
digital reference service during a specific 
time period. 

5. Awareness. At least XX% of a specific 
target population (specific population 
group, e.g., graduate students) will be 
aware that the library provides (specific 
type of) digital reference services. 
Awareness is defined as the population 
user group’s knowledge that the service 
exists. The process for collection of this 
data is to conduct a survey of the target 
population to determine the extent to 
which the population group is aware of 
the service. 

6. Cost. The cost per digital reference 

transactions will not exceed $XXXX. 

Using the procedures described earlier in 
this manual, library staff will compute the 
cost per digital reference transaction. The 
average cost should be based on a sample 
conducted over an acceptable period of 
time. The average cost per transaction is 
then compared against the agreed upon 
quality standard. 

7. Completion Time. The average 

completion time for digital reference 

transactions will be X amount of time. 
This quality standard will vary greatly in 
each situation for a number of reasons 
including types of reference questions 
answered, difficulty of reference questions 
answered, and hours of operation. This 



quality standard should be determined 
very carefully keeping in mind that the 
fastest service may not always be the best, 
but that users do expect responses in a 
timely fashion. 

8. Accessibility. Accessibility refers to a 
wide range of factors including ease in 
finding and vising the service and 
availability of adaptive software for users 
with special needs. Quality standards such 
as “A user will be able to access the 
service with only X number of clicks from 
the library home page” or “Every user 
with special visual needs will be able to 
‘read’ every word and link on the digital 
reference homepage using voice reading 
technology.” 

The above quality standards suggest the 
way in which the various statistics and 
measures outlined earlier in the manual can be 
translated into quality standards. Those 
proposed here are examples; library staff may 
wish to develop additional or substitute 
quality standards. 

Determining the “Right” Standard 

This is no “correct” standard for any 
specific digital reference service. The correct 
standard will depend on the goals and 
objectives of the library, the amount of 
resources that can be committed to reaching a 
particular standard, local situations affecting 
digital reference services, and the relative 
importance of one quality standard versus 
another. For one library, an awareness level of 
digital reference services of 30% among 
faculty (for example) may be acceptable; for 
another, the standard might be 60%. 

Typically, when setting quality standards, 
a group of library staff and administrators will 
first benchmark the existing performance for 
a particular proposed criterion (awareness, 
satisfaction, etc.). For example, they would 
collect data to determine what the existing 
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correct answer fill rate is. If they determine 
that the existing rate is a 45% correct answer 
fill rate, then the next step is to discuss if this 
is an acceptable level for the standard. Given 
the goals, resources, and unique situations 
affecting digital reference at a particular 
library it may or may not be acceptable. 

As an external check on the 
appropriateness of a standard, library staff can 
meet with users of the service and obtain their 
input as to an appropriate level for the 
standard. Library staff might be quite 
surprised to learn that users of the digital 
reference service may have very different 
levels of expectations for the quality of a 
service. If significantly different levels of 
expectations in level of quality between 



library staff and users exist, additional 
discussion will be necessary. The important 
point, however, is that library staff and 
administration agree on what the correct 
answer fill rate standard should be and then 
strive to meet that standard. 

Whatever quality standards are developed 
for a particular digital reference service, it is 
important to regularly check the library’s 
performance on that standard over time. Is the 
library meeting the standard? If not, why not? 
If the standard is being met should the quality 
standard ,be increased to promote better 
service and performance? These and related 
questions should be ongoing. 
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Async 

Async refers to transactions/communications 
that occur between the staff and user that do 
not occur in real time. There will be a gap in 
time between such communications. An 
example of an async transaction in a 
traditional library setting would be the 
submission of a print subject request form. In 
the digital reference setting an async 
transaction would include the submission of 
an email or online webform request to the 
service. 

Digital Reference Answer 
For the purposes of this manual a digital 
answer is the aggregate of responses sent in 
response to a digital reference question 
submitted by the user. The aggregate of 
responses is counted as one digital answer. A 
digital answer must be sent entirely via 
computer or digital means. A digital reference 
answer also includes referrals to other contact 
points and directions to resources that would 
enable the user to locate the answer on his/her 
own. 

Digital Reference Question 
For the purposes of this manual a digital 
question is a question that is submitted to a 
digital reference service using digital means 
(i.e., must be transmitted via computer using 
such methods as email, webform, chat, or 
interactive video). Questions submitted by fax 
or phone are not considered digital questions. 
Questions that are received digitally but 
responded to with non-digital means are not 
considered a digital reference question for the 
purposes of this manual. 

Digital Response 

A digital response is any communication that 
is sent to the user electronically in response to 
a digital reference question. A digital 



response will not necessarily include the 
answer to the reference query, and may in fact 
be an acknowledgement of receipt of the 
question, a request for more information, a 
request for clarification, or a message asking 
if what has been found is satisfactory. 

Format of Answers 

Format of answers includes: all full-text 
material, combination of full-text and citation 
or abstracts, journal articles only, web-pages 
only, complete literature searches, analysis of 
materials located, placement of request for 
books, or delivery of actual books. This 
measure can also reflect the expectations of 
the user in terms of physical format, i.e., html, 
pdf, ASCII or plain text, or various word 
processing formats (such as MS Word, Word 
Perfect). Additionally, the expectation of the 
user may include that the digital reference 
service staff will provide conversion of 
material from one format to another. 

Logs and Reports - Electronic 
An electronically generated log or report is 
one that is provided with weblog software 
(such as Webtracker or Webtrends) or digital 
reference software (such as 24/7 or LSSI’s 
VRS), or custom designed databases (using 
such products as MS Access). 

Logs and Reports - Manual 

A manual log or report is one that is created 
by the user. Entries are made manually into a 
log and reports based on the logs must be 
generated manually. 

Real-time _ 

Real time refers to a description of a 
transaction or communication that occurs 
between staff and the user without any 
intervening time between the points of 
communication. 
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Interactive Video 

Interactive video digital reference is the 
ability of the staff and user to view each other 
using video signals transmitted through the 
computer. This type of digital reference 
transaction seeks to simulate a face-to-face 
transaction that would occur in a physical 
library setting. Advantages are that the staff 
and user can read each other's body language 
and gather much information regarding 
attitude and demeanor from the "face-to-face" 
setting. Some limitations to the 
implementation of interactive video are that it 
requires a great deal of bandwidth, special 
equipment and software for both user and 
staff. 

Sync 

Sync refers to transactions/communications 
that occur between the staff and user in real 
time. There is no delay or gap in time 
between the receipt of communication and 
response. Examples of a sync transaction in 
the traditional reference area are the face-to 
face reference interview and a telephone 
conversation. In a digital setting sync 
transactions include chat and interactive 
video. 

Voice over IP (V oIP) 

Voice over IP (VoIP) allows for the staff and 
user to talk to and hear one another over an 
Internet connection. The primary advantage 
of voice over IP is the ability of the user and 
staff member to engage in a real time 
conversation. This saves time in the reference 
interview process, avoids possible long 
distance phone charges, and allows for both 
the user and staff member to detect inflections 
in tone of voice that may impact the reference 
transaction. The primary disadvantages are 
that VoIP is bandwidth intensive and that 
transmissions often experience static, or are 
unclear. 



Types of Reference Questions 

Bibliographic 

A bibliographic reference question is one that 
relates to any aspect of authorship or 
publication of a work. Bibliographic reference 
queries include verification of a citation, 
names of authors, information about works in 
a series, edition information, copyright 
information, etc. 

Instructional 

An instructional question is one in which the 
user asks for assistance in using electronic 
resources that may be available to them, and 
that may provide the answer to another 
reference question. Examples of instructional 
questions include requests for information on 
how to construct a search statement in an 
online periodical database, how to search the 
online catalog (OP AC), how to request books 
and other materials from the catalog, how to 
limit searches by domain in a particular 
search engine, and how to use Boolean Logic. 

Literature search 

A literature search is a request for all of the 
published literature on a given topic or by a 
given author. The literature search may be 
limited by such factors as date, place of 
publication, peer-reviewed journals only, etc. 

Other 

For the purposes of this manual, “other” 
refers to those questions that are wit hi n the 
scope of the service but do not fit into any of 
the other categories. Digital reference 
questions that fit more than one category 
should be classified as “other.” 

Out of Scope 

An out of scope question is one that will not 
be answered by the digital reference service 
because it does not meet the criteria set by the 
service for provision of an answer. Out of 
scope questions are often referred to another 
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service within the organization or to an 
outside agency or service. 

Reader's Advisory 

A reader's advisory question refers to requests 
for information regarding material they would 
like to read. Reader's advisory questions often 
take the form of asking for similar books by 
plot, other books by an author, other books in 
a series, availability of works in a specific 
format (e.g., large type, book on CD or tape) 
works in a particular language, or information 
about the background of a particular book. 

Ready Reference 

Ready reference (or quick fact) questions are 
those that usually have a single, finite answer. 
The answers can generally be found in 
common reference works such as almanacs, 
encyclopedias, directories, dictionaries, 
atlases, thesauri, and factbooks. Some 
examples of ready reference questions 
include: What is the population of Indonesia? 
What is the capital of Brazil? On what date 
did Queen Victoria die? How do you spell 
“symbiotic”? It should be noted that many 



digital reference services will only answer 
ready reference type questions. 

Research or Subject Request 

A research question is one that requests a 
variety of information on a particular topic. 
The research question will most likely have 
many components to the answer (i.e., articles 
from journals, book citations, essays, 
statistics, raw data) and the answer may 
consist of responses sent in many formats 
(emailing of full-text articles or citations, 
pushing websites, documents or spreadsheets, 
image files, video clips, etc.). 

Technical 

A technical question is defined a as reference 
question in which the user asks for assistance 
in the use of the technology required to access 
the digital reference service or other aspects 
of accessing the library’s or organization’s 
website. These questions can include: How do 
I download Adobe Acrobat Reader? How do I 
open attachments? How do I install chat 
software? 
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APPENDIX I 



SAMPLE FORMS, REPORTS, LOGS, WORKSHEETS AND 

SURVEY INSTRUMENTS 



A. Internet Public Library “Ask A Question” digital reference web submission form 

B. Sample Webtrends Report - General Statistics 

C. Sample Webtrends Report - Activity by Day of the Week 

D. Sample Webtrends Report - Activity by Hour of Day 

E. Sample Summary Report from 24/7 Reference Software 

F. Sample User Satisfaction Transcripts and Reports from LSSI’s Virtual Reference 

G. Sample Chat Transcription form LSSI Virtual Reference 

H. Sample Digital Reference Transaction Record 

I. Digital Reference Question Transaction Log - Email 

J. Digital Reference Question Transaction Log - Real Time Sessions 

K. Total Reference Activity Log -Daily /Two Week Period 

L. Digital Reference Completion Time Calculation Worksheet 

M. Sample Digital Reference Data Collection Worksheet - Compilation 

/ 

N. Type of Digital Reference Question Received 

O. Sources Used Per Question 

P. Log Analysis - Number of Digital Reference Sessions - Real Time 

Q. Log Analysis - Usage by Day of the Week / Usage by Time of the Day 

R. Log Analysis - User's Browser / User's Platform 

S. User Satisfaction Measures - Expectations for Service Sample Questionnaire 

T. User Satisfaction Measures - Sample Survey/Questionnaire Questions 

U. Cost Worksheet 

V. Cost - Electronic Resources Prorate Worksheet 

W. Cost of Digital Reference as Percent of Total Reference / Total Library Budget 

X. Staff Time Expended - P ercent of T ime Overseeing T echnology 

Y. Staff Time Expended - Percent of Time Assisting Users with Technology 
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SAMPLE 

A. Internet Public Library “Ask A Question” digital reference web submission form 
http://www.ipl.org/ref/QUE/RefFormQRC.html 



•'jjll'L Ask A UuriI it 



rr - D»r;<*tnscap«r Unhne - 1 -OUU-GC V-t HDb 












nsra i 



http:/A»»w ipto«gAtf/OUE/R ^ on^HCl»wjr 



ml 



ie Internet Public Library 

EPL Ask A Question Form 



Wb acre not able to perform lengthy research. However we can provide brief answers to factual questions or suggestions for locations and sources 
which might help to answer your question. 

P T.EAX fi! BEAD ! About tho Iwf, AyV-AQueition Service and The Tt*i.*y Privacy Statement 

Before you ask a reference question, please check to see if your question is in the Frequently Asked Reference Questions Hst. You could save yourself and 
us. a lot of time. 



j What is your name? 1 1 

What is your email address? } 1 

If you don't give us your correct, complete Internet email address (example: fiuggly@AoL com), we can't send you an answer to your question. * *AOL 
users** - If you have parental or mail controls turned on. please add iplrefi^ipL org to your allowed mail list — otherwise we can't tend e-mail to you I 
Where do you live? (CnyfState/C oxmtry) | j 

We can usually help you better if we know where you Hve. and how far away you are from the resources we may recommend to you. 



1 1 won't treed this information after - . [[ 



j (mm/dd/yy) 



(Note - we cannot help you if you need an answer in < 3 days. CEck here if you are in a hurry ) 
[pH Tho Subject Area of tho Question: (click to see list — choose one) 

| (None Selected) jj] 



ill 



|[4 j Please tell us your question. 

A human being will read your question — please use complete sentences I 

The more you tell us. the better our answer will be. What do you already know about your subject or question? 




How will you use this information? Why are you asking your question? 

If you're just curious, that's ok, but it really helps librarians to know this parti Sometimes we can use our subject knowledge and imaginations to think of 
other places to look for answers and information, if we know how you will use it or what you want to get out of the answer. 
; 1 



Will you use this information for a school assignment? C Yes C No 
Are you; D A librarian? □ A teacher? O A businessperson? 

|[6| Type of answer preferred; (choose one of die following) 

C A brief factual answer to your question 
C Some ideas For sources to consult for exploration: 

□ interact sources □ Print sources G I don't care which kind 



BbS'l COPY AVAILABLE 
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SAMPLE 

A. Internet Public Library “Ask A Question” digital reference web submission form Continued 
http ://www.ipl.org/ref/QUE/RefF ormQRC.html 



..lock ' B«jrT^TT'|i'T^Vy ^-vo' .t -t '• ‘ t 



I A^dreii httoc//ww».cto cgAtf/QUE /H off 0CnqRC.hCrt ' 



£jjlPl Ask A Question form - Dreomscape Online * 1 '8118-GET- 11195 






mm 



|[fij Type of answer preferred: (choose one of the fofiowing) 

C A brief factual answer to your question 
C Some ideas for sources to consult for exploration; 

G Interact sources G Print sources G I don't care which kmd 

Sometimes the information you want crit available on foe Internet, but might be available through a library near you. We can almost always get you started, 
at least 



|[p| Sources Consulted: 

Please Hst any places on foe Net or off that you’ve already checked regarding your question. We don’t want to duplicate your attempts. Don't forget to try 
using our Ready Reference Collection and your local library to answer your question. 

3- 



|pSN| SEND 111 

Reminder: Please take a moment to re-check foe e-mail address you are submitting to us, since it is impossible for us to communicate with you unless it is 
correct Also, if you have not read our Privacy Statement, please do so. Thanks! 



House confirm your email a d dress J - 



n SuMj Questioivtp^ f 

If you have problems using this form, you can also submit a question by e-mail For instructions, consult die E-Mail Guidelines . 

Return to Ask a Question j IFL Reference Center | jPL Lobby 

the Internet Pubhc Library - = ♦ httpJhrmv.ipLarg/ - = - rpi@ipLorg 
Last updated Feb 26, 2002 



u 




BEST COPY AVAILABLE 
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SAMPLE 

B. Sample Webtrends Report - General Statistics 





The Visits graph displays the overall number of visits to your Web site. The General Statistics table 
provides an overview of the activity for your Web site during the specified time frame. 



Visits 




07)01 07/15 07/29 00/12 00/26 09/09 09/23 



Sun 07/01 /2001 - Sun 09/30/2001 (1 Quarter Scale) 



General Statistics 


Hits 


Entire Site (Successful) 


552 




Average Per Day 


6 




Home Page 


N/A 


Page Views 


Page Views (Impressions) 


552 




Average Per Day 


6 




Document Views 


0 


Visits 


Visits 


262 




Average Per Day 


2 




Average Visit Length 


00:04:04 




Median Visit Length 


00:00:43 




International Visits 


0% 




Visits of Unknown Origin 


16.79% 




Visits from United States 


83.2% 


Visitors 


Unique Visitors 


160 




Visitors Who Visited Once 


140 



BEST COPY AVAILABLE 
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SAMPLE 

C. Sample Webtrends Report - Activity Level By Day of Week 




This page shows the activity for each day of the week within the reporting period. Unsuccessful hits are 
not included. 



Activity Level By Day of the Week 




Activity Level by Day of the Week 

- - ' .... 


0.! '' V 








ft] 


-«j>r 


* K 

Day 

.. .i-MiMi&t 


A ■ T ?!§!» 


! H,ts 1 


% of Total ) 
Hite ! 
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1 


Sun 






39 
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20 


2 


Mon 
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63 


3 


Tue 
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19.56% 


52 


4 


Wed 
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38 


5 


Thu 






75 


13.58% 


32 


6 


Fri 






74 


13.4% 


39 


7 


Sat 






31 


5.61% 


18 
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Weekdays 
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87.31% 
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Total Weekend 


.. — 




r~^ 7 °i 


F 12.68%} 
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SAMPLE 

D. Sample Webtrends Report - Activity Level by Hour of Day 



Activity Level By Hour of the Day 




Hours 



Activity Level by Hours Details 

'<r T. “s 

■*<W - ' «?*ltf.**4* 


wmwB 


immm 


if*# 


mm 

Vo of Total ts . 


:#of 

Visits 


00:00 - 00:59 






12 
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6 
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4 
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0 


0% 


0 
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3 
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0% 
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19:00-19:59 






35 
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19 
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29 
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19 
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29 
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18 
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14 


2.53% 


7 
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8 


1.44% 


5 


Total Visitors during Work Hours (8:00am-5:00pm) 
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SAMPLE 

E.Sample Summary Report from 24/7 Reference Software 



Library Summary Reports 



& REFERENCE 



Name 


; Call Resolution Codes ] 

Cq^X^ERINXFERQUTTESTWCSrADANWCS-GONCWCS-CtQSTECHPROSbUTSCOPEOTHERNQT SPEC.TOTAL 





! o | 


0 


0 


! o 




0 


L 1 


0 


0 




0 




0 | 


! • 5 


6 


cbonnetil 


o 


0 


“ 6 " 


! ’ l ! 




0 


i o " 


"”i“ 


0 




0 




1 b 1 


[ o 


2 


steal 


! o i 


0 


o 


0 




o 


d 


0 


2 




0 




1 0 1 


0 


2 


C,gtea 


1 5 j 


1 2 


o 


0 




o 


0 


0 j 


i o 




0 




1 0 


0 


7 


dans 


3 


! 1 


o 


! o 1 




o 1 


1 o 


X 


| 2 




i 




: o i 


! X 


9. 


ieff s 


1 


0 


o 


0 




T 


n o 


o 


o 1 




0 




i o s 


1 o 


x“ 


3annifer 


; ' 9 j 


1 


' o 


'1 0 j 




o ] 


1 3 


o 






2 




o 1 


o 


. 15 ' 


Jennifer o 


! 1 1 


o 


o 


1 o 1 




d 


i d 


o 


0 




0 




1 0 I 


o 


1 


3?ifian 


! 3 


i 


o 


1 o ! 




o 


r o 


0 


i o 




0 




! o J 


2 


6 


3im ... 


’ 9 ! 


o“ 


o " 


T'"o 1 




o ] 


[ o 


0 


f o ~] 




o 




; 0 1 


i 2 


11 


latoZZ Z] 


: 5 1 


1 l 


o 


1 o ! 




6 


[_X 


0 


j"“ "o" ! 




i 




0 


x 


8 


Kate 


5 


! 0 


’ o~~ ‘ 


1 "o 




0 : 


i i 


” d ! 


[ 0 




o 




0 


r o 


'6 


Melissa 


L i | 


! X 


o 


i o 




0 


! o 


0 


0 




6 




0 


o 


2 


mgaggibrgcht 


i i 


0 


0 


T o ! 


0 


I 0 


0 


1 0 


0 


[ 0 


0 


Trf 1 


Nanc? 


1 ! 


1 


0 


! z"l 




0 


o" 


0 1 


i 1 i 




o 




t 0 


0 


5 


Pauline 


: 4 


! i"' 


o’- - 


i on 




0 


r " o 


o“ " | 


i 




o " 




0 


r x 




£asbsl..II"I] 


2 


0 


0 


0 




o 


0 


0 


3 " 1 




jjj 




i o ; 


o 


S “ ' 


robert 


1 3 i 


0 


0 


0 




d 


2 


6 


2 




0 




1 0 


0 


7 


Em 


| 4 j 


0 


0 


! 0 : 




o 


0 


o 1 


0 




0 




0 


0 


4 


ikark 


i i 1 


0 


0 


0 




0 


0 


o 


o 1 




0 




0 


0 


•x 


Steve 


6 1 


cf 


oT 


i 1 1 


0 


0 


0 


o • 1 


0 


0 


3 


ib™ 


im 


64 


9 


0 


1 4 1 


0 


7 


2 


li 


4 


0 


15 


116 



f 

Login Main Page Reports Menu Report 

' / 



o 

ERIC 



80 



BEST copy available 



72 




Statistics, Measures, and Quality Standards for Assessing Digital Reference Library Services: Guidelines and Procedures 



SAMPLE 

F. Sample User Satisfaction Transcripts and Reports from LSSI’s Virtual Reference 




'3 LSSI Repo rts ; ^ Mi cro^^ r e c r „ 









Rle igfit Favorites lools fcjdp * 






Google r\ li^l^iSearch Web'C^ 


^ O Pageirife z \ * 






LSSI Report 


i 


PaSS™ Survey Details 




$ 


Statement 


Answer 




Which information source do you use most often? 


Intemet/Web 




How quickly did the virtual librarian understand your question? 


7 


m 


How easy was it for you to understand the virtual librarian s answers? 


5 


.... 

i 


How friendly was the virtual librarian? 


6 




How helpful was the virtual librarian? 


6 


1 

; ‘ 


How prompt was the virtual librarian in answering your question? 


7 


How satisfied were you with the virtual reference experience? 


7 




What percentage of the time would you use us to answer your future 


6 


: 

■M- 


questions? 




>i 1 


Comments: 


;■ 1 
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SAMPLE 

G. Sample Chat Transcription from LSSI Virtual Reference 



TAMSIN Bolton’s Virtual Reference Transcript from the University of Winnipeg Library 
Winner LSSI’s Samuel Swett Green Award for Best Virtual Reference Transcript 5/15/02 
All personal information has been removed to protect the privacy of the patron 



Submit No: 00004 
Name : 

Library: 

Position : 

TelephoneNumber : 

PatronWaitingTime : No wait 

NumberOf Inquirers : 1 

Transcript: General Chat Info 

Chat start time 11:36:39 04/15/2002 

Chat end time 12:27:36 04/15/2002 

Duration (actual chatting time) 00:50:43 

Chat Transcript 

info: Please wait for a site operator to respond, 
info: You are now chatting with 'TAMSIN' 

TAMSIN: Hi there, how can I help you? 

VISITOR: Hi, TAMSIN, my name is * and i was just chatting with * about XXXX 
[database vendor name], i don't know how to access the articles she helped me 
find 

TAMSIN: okay, did she e-mail them to you or are you just trying to . get 
At them in XXXX [database vendor name]? 

VISITOR: i'm trying to access them from home, she offered to email 
them, but i'm trying to familiarize myself with the process to use it as a 
resource, when i click on full text, etc, it doesn't bring up the articles, 
can i not view them and print them? 

VISITOR: maybe i don't have the proper software, is that possible? 

TAMSIN: yes you certainly should be able to view them. Are you by any 
chance clicking on a link that says "full page image"? 

VISITOR: when i tried to download the article from its current 
location, it said there was no path, yes i did try full page image, what now? 
TAMSIN: Well, full page image requires that you have software called 
"acrobat reader" installed on your computer. If you are not able to read. the 
file it is possible your don't have acrobat reader- you can easily install it 
right now if you want though 

VISITOR: i don't know if it will fit, does it require a lot of space? 
i'm so ignorant of the techincal world, i don't even know how to check how 
much space i have 

TAMSIN: umm, it shouldn't require too much space. But it can sometimes 
take a bit of time to download. Do you have a high speed internet connection? 
VISITOR: i 'm connected to the u of w free internet service, on a 
Compaq 586 that was built in 1995! 
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SAMPLE 

G. Sample Chat Transcription from LSSI Virtual Reference Continued 



VISITOR: the time doesn't really matter, i just don't know if it will 
work 

TAMSIN: oh my:-), well, that might take a bit of time then. If you 

aren't concerned about time then you could try to download it but, like you, 

I am not sure how well the program would run with your computer 
TAMSIN: was "full page image" the only full text link you saw? 

VISITOR: hang on, i'm just checking 

TAMSIN: because XXXX [database vendor name] does sometimes offer different 
formats of full text 

TAMSIN: full page image" is the only version that requires seperate 
software 

VISITOR: some of the articles have different options, some have all 
"full page image" and "full text" and "XML full. text", others only have two 
of those options, still others a combination 
VISITOR: which one should i choose? 

TAMSIN: if you click on "full text" you should see the text on your 
Screen without any seperate software 

VISITOR: i'm going to try that, can you hang on while i try that? 

TAMSIN: sure thing, I'll be right here 

VISITOR: it takes me to a page that says refine search result list. 

This is where i don't know what to choose 

VISITOR: there is an option that says "go to citation", is' this what I 
click on? 

TAMSIN: Are you looking at your list of results right now? 

VISITOR: no, i clicked on one of my results and it took me to this 

Results page, i'm going to try clicking on the citation, i'll be right back 

VISITOR: nothing is happening when i click on citation 

TAMSIN: okay- do you think you could tell me the name of the article 

You are at? 

VISITOR: it wasn't anything specific, just an article that had the 
Option of full text, i have to. check what it was . . . 

TAMSIN: Well, from here, when I click on full text I see the article 

on my screen and that link "go to citation" is in the top right corner.. But 

you are not seeing the full text, is that right? 

VISITOR: yes 

TAMSIN: hmmm, this is a pickle... 

VISITOR: i don't see the article though 
VISITOR: that's exactly what i said to * 

TAMSIN: Let's just test something out- pick any article from your list 

that says it has full text. I'll look at it here at see if I have any trouble 

VISITOR: it's the attack on human rights, by Michael Ignatieffk, 

Foreign Affairs, nov/dec, 2001 
TAMSIN: okay- I'll check it out 

VISITOR: that's supposed to read Ignatieff without the "k" 

TAMSIN: Okay, I've got it. So if you click on the title of that 
Article and scroll down, you don't see the article? Am I right? 

VISITOR: you're right 
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SAMPLE 

G. Sample Chat Transcription from LSSI Virtual Reference Continued 



TAMSIN: yikes ! I think that pickle comment I made may be an 

understatement:-). You should be able to see the full text on your screen and 
I am really not sure why you are not. 

TAMSIN: I know you wanted to figure this out yourself but I am still 
Very willing to e-mail you some articles if you want 

TAMSIN: and in the meantime I can try to figure out why you may not be 
able to view the full text 

VISITOR: you’re awesome, i guess i will have to take some time after 
The term to get this whole cybrary thing figured out. do you have an email 
address and i can mail you a list of what i need? i need to take the time to 
choose the articles, twelve in all. meanwhile, should i try to down load 
acrobat, and how do i do that? 

TAMSIN: sure, you can e-mail me at t.bolton@uwinnipeg.ca - and I'll 
Send you to the page to get acrobat reader right now 
VISITOR: how are you sending that? 

TAMSIN: one sec.- it should appear on your screen like magic... 

VISITOR: you are so funny! 

TAMSIN: http : //www. adobe . com/products /acrobat /re ads tep2 . html 
TAMSIN: ta-da 

TAMSIN: do you see the instructions? 

VISITOR: you are so magical, it's here and you are fabulous, thanks 
For your email, i will chat with you later, i can't thank you enough for 
taking the time, you guys are all really awesome, so awesome in fact, you can 
have my firstborn, she's driving me crazy 

TAMSIN: okay, * is here with me and she wants to test something out 
•With you- do you have a minute? 

VISITOR: i have lots of minutes 
TAMSIN: The Attack on Human Rights. 

TAMSIN: when you click on the title can you see the fulltext? 

VISITOR: i'm stuck in XXXX [database vendor name], then only thing i see 
now is return to Top and when i click, it won't do anything, should i try to 
get back in, i'm turning in to a nightmare, aren't i? 

VISITOR: ok, i misunderstood, i will click on your link 
TAMSIN: ok 

VISITOR: now i'm on XXXX [database vendor name] publishing login page and 
it is asking me for an id and password 
VISITOR: what now? 

TAMSIN: Okay, our test didn't work. Let's just stick to our other 

plan- you e-mail me with the articles you need and I'll e-mail you back the 

text. Sound good? 

VISITOR: sounds good, i'm also going to fool around with the acrobat 
Thing and try a couple of things from here, i'll email you to let you know 
where i'm at. thanks again, talk to you later 
TAMSIN: sure thing- good luck and yes, talk to you later:-) 
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SAMPLE 

H. Sample Digital Reference Transaction Record 



NOTE: This sample form is intended to serve as basis for developing a form that is most useful to your library or 
organization. It is not intended that this form be copied and used as is. Some of the features and questions on this 
form will be useful in some situations and not in others. It is not expected that you will use all the elements of this 
form. It is comprehensive so that you may see what questions you might use in creating your own form. The form 
you develop may be a print form or developed into a database or spreadsheet. 



Digital Reference Question Transaction Record 

Instructions: Complete a separate form for each digital reference question received . 

Date Question Sent from User . Time Question sent 

Day and Date Received : Time Received 

Date Completed Time Completed 

Total Transaction time 



Question received by: (check appropriate media) Staff member 

email chat webform VoIP Interactive video instant message 

User Information 

User Name 

User contact information/email address 



User demographic information (enter appropriate demographic information for your situation): 

Is this patron a repeat user? Y N Reason for repeat 
use . 

User's Question Information 

User’s Reference Question 

Additional information regarding question: 



Type of reference question (i.e. ready reference, instructional, subject request? etc): 

Date information needed by RUSH? 

Delivery preference? 
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SAMPLE 

H. Sample Digital Reference Transaction Record Continued 



Does the user have any requirements regarding format of answers or responses? Y N 
If yes please specify (.doc .pdf, .xls, etc.): 

Magazine or journal articles only? Y N Full - text only? Y N 

Citations/Abstracts acceptable? Y N Web info acceptable? Y N 

Email attachments OK? Y N 

Does the patron have any specific requirements/limitations regarding type of website or other resources 
(i.e. peer-reviewed, specific domains, government information, primary resource materials only, etc)? Y 
N 

Specify: ■ 

User's Browser User's Platform 



What other sources, if any, has the user tried? 



Staff Responses/Answer 

Were you able to answer the user's question? Y N 
Describe the nature of the answer provided to the user: 



What resources did you use to answer this question? (for checklist, check all that apply) 



Free_ Staff-Created Proprietary Traditional Other 

Did you refer this question? Y N If Yes, Internal? External? 

Where did you refer the user? 

Reason for referral 

Would you have been able to answer this question? Y N 

If yes, why did you give a referral? 

Total digital responses Were any non-digital responses made? Y N 

NOTE : If non-digital responses were made-do not include this question as a digital reference 
question 
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SAMPLE 

Digital Reference Data Collection Worksheet 

I. Digital Reference Question Transaction Log - For libraries providing email digital reference 
service 



This transaction log is for use primarily by those libraries that provide both traditional and digital reference 
service (where the digital reference service is primarily email based). Use this log in conjunction with the 
Digital Reference Transaction Record to keep track of all digital reference transactions. 

Name of Staff Member 



Question 


User 


Date/Time 

Initial 

Email 

Sent 


Date/Time 

Initial 

Email 

Rec’d 


Date/Time 

Digital 

Reference 

Transaction 

Completed 


Total 

Time 


Type 


EXAMPLE 

Latest population of Bolivia 


Amgad 

E. 


7/5/02 
10:05 a.m. 


7/5/02 
10:12 a.m. 


7/5/02 
10:31 a.m. 


26 

minutes 


ready 

reference 


Author of the short story 
"Midnight Express" 


Sebastian 

S. 


7/5/02 
10:09 a.m 


7/5/02 
10:32 a.m 


7/5/02 
10:41 a.m. 


31 

minutes 


bibliographic 
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SAMPLE 

Digital Reference Data Collection Worksheet 

J. Reference Question Transaction Log - For libraries providing real-time digital reference service 



Question 


User 


Date/Time(s) 
Initial Session 
requested 


Date/Time(s) 
Initial Session 
honored 


Date/Time 
Final Session 
Completed 


Total 

Time 


Type 


EXAMPLE 
How lighthouses 
work 


Anna Maria 
L. 


9/16/02 
1:02 p.m. 


9/16/02 
1:03 p.m. 


9/16/02 
1:15 a.m. 


13 

min 


ready 

reference 


competency 

modeling 


Frank D. 


9/16/02 
2:16 p.m. 


9/16/02 
2:16 p.m 


9/16/02 
3:14 a.m. 


58 

min. 


research 




















































































- 






























TOTAL 


1 f $ j.?y» T* 

sj- -t; • •* 


jf >* ( = 

$als, k" : 

U & ■$ : f, *£ £. ! 


■*■%* *f «*<»## i? t J | 
;«a 4 4 * “ifr#' ** ^ 1 “! 

&& 4 «#%># £».-4 i 1 1 

i cV„ f 4 ■ $ U.f 4 ? * .5 

f $$$ v |i >• 


’■A*# 4 ■* 

■> * 44 . «ri 

(*•*4 ** $••; v 5 '- *: 


71 

min. 


>-*?>*$»■ ? •% §!• W# * 

- % setyf* ■(•*» ,<1 ’ 

*••«* - a*" # :! <^ : 



Statistics, Measures, and Quality Standards for Assessing Digital Reference Library Services: Guidelines and Procedures 



SAMPLE 

Digital Reference Data Collection Worksheet 

K. Total Reference Activity Log - Daily / Two Week Period 



Total Reference Activity Log - Daily 



Date 



Staff Member 



TRANSACTION 


Mark each of transaction in the space below 


TOTAL 


Digital 


m llll 




Hybrid * 


nun 




Traditional 


m 





NOTE: Hybrid Reference Transactions are those transactions which are submitted digitally but 
which are responded to or answered by non-digital means as defined in Measure 4. 



Total Reference Activity Log - Two Week Period 



Date 


Digital 


Hybrid* 


Traditional 


Total 
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NOTE: Hybrid Reference Transactions are those transactions which are submitted digitally but 
which are responded to or answered by non-digital means as defined in Measure 4. 



